[jira] [Commented] (HDFS-11145) Implement getTrashRoot() for ViewFileSystem
[
https://issues.apache.org/jira/browse/HDFS-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678727#comment-15678727
]
Hadoop QA commented on HDFS-11145:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11145 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839676/HDFS-11145.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 9af403cd995e 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7584fbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17615/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17615/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17615/console |
| Powered by | Apache
[jira] [Updated] (HDFS-11145) Implement getTrashRoot() for ViewFileSystem
[ https://issues.apache.org/jira/browse/HDFS-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HDFS-11145: -- Status: Patch Available (was: Open) > Implement getTrashRoot() for ViewFileSystem > --- > > Key: HDFS-11145 > URL: https://issues.apache.org/jira/browse/HDFS-11145 > Project: Hadoop HDFS > Issue Type: Task >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Labels: viewfs > Attachments: HDFS-11145.01.patch > > > ViewFileSystem doesn't have the custom implementation of > FileSystem#getTrashRoot(Path) yet and hence irrespective of Paths passed in, > ViewFileSystem always returns the user specific .Trash directory. > ViewFileSystem should implement getTrashRoot(Path) and delegate the call to > the respective mounted file system which can then examine about EZ or other > criteria and return a proper Trash directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11145) Implement getTrashRoot() for ViewFileSystem
[
https://issues.apache.org/jira/browse/HDFS-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11145:
--
Attachment: HDFS-11145.01.patch
Attached v01 patch which takes care of the following
* Implemented {{getTrashRoot(Path)}}, {{getTrashRoots(users)}}
* Updated ViewFileSystemBaseTest base class test to verify {{getTrashRoot}} for
all filesystem derived tests
* Updated TestViewFileSystemHdfs to verify {{getTrashRoots}} under EZ settings.
> Implement getTrashRoot() for ViewFileSystem
> ---
>
> Key: HDFS-11145
> URL: https://issues.apache.org/jira/browse/HDFS-11145
> Project: Hadoop HDFS
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Labels: viewfs
> Attachments: HDFS-11145.01.patch
>
>
> ViewFileSystem doesn't have the custom implementation of
> FileSystem#getTrashRoot(Path) yet and hence irrespective of Paths passed in,
> ViewFileSystem always returns the user specific .Trash directory.
> ViewFileSystem should implement getTrashRoot(Path) and delegate the call to
> the respective mounted file system which can then examine about EZ or other
> criteria and return a proper Trash directory.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11159) Add reencryptEDEK interface for KMS
[
https://issues.apache.org/jira/browse/HDFS-11159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678418#comment-15678418
]
Hadoop QA commented on HDFS-11159:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange} hadoop-common-project: The patch generated 1 new
+ 42 unchanged - 2 fixed = 43 total (was 44) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
4s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
13s{color} | {color:green} hadoop-kms in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11159 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839664/HDFS-11159.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 17e44b253219 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7584fbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17614/artifact/patchprocess/diff-checkstyle-hadoop-common-project.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17614/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-common-project/hadoop-kms U: hadoop-common-project |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17614/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678377#comment-15678377 ] Xiao Chen commented on HDFS-10899: -- Thanks a lot [~andrew.wang] for the great reviews and also [~dilaver] for offline suggestions! Still wip, but attaching for review. What's changed in wip2: - Split kms related change to a subtask. - Addressed most comments. Except some items left as todo: : command usage, writelock when talking to kms. Also structuring as iterator/visitor part isn't done, but inode based traversal looks better now. TODO items: - cancel / verify commands - better metrics - lock tuning and potential batching communication with kms, based on test results. - some additional tests for correctness (raised by Dilaver): -- rm file while its EZ is being reencrypted -- mv file while its EZ is being reencrypted -- rm & recreate file while its EZ is being reencrypted -- append to file while its EZ is being reencrypted > Add functionality to re-encrypt EDEKs. > -- > > Key: HDFS-10899 > URL: https://issues.apache.org/jira/browse/HDFS-10899 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt > edek design doc.pdf > > > Currently when an encryption zone (EZ) key is rotated, it only takes effect > on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key > rotation, for improved security. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11154) Block Storage : store server state to persistent storage
[
https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678373#comment-15678373
]
Hadoop QA commented on HDFS-11154:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
6s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
7s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
17s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 10
unchanged - 0 fixed = 13 total (was 10) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Found reliance on default encoding in
org.apache.hadoop.cblock.CBlockManager.createVolume(String, String, long,
int):in org.apache.hadoop.cblock.CBlockManager.createVolume(String, String,
long, int): String.getBytes() At CBlockManager.java:[line 226] |
| | Found reliance on default encoding in
org.apache.hadoop.cblock.CBlockManager.deleteVolume(String, String, boolean):in
org.apache.hadoop.cblock.CBlockManager.deleteVolume(String, String, boolean):
String.getBytes() At CBlockManager.java:[line 237] |
| | Found reliance on default encoding in
org.apache.hadoop.cblock.CBlockManager.readFromPersistentStore():in
org.apache.hadoop.cblock.CBlockManager.readFromPersistentStore(): new
String(byte[]) At CBlockManager.java:[line 255] |
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.hdfs.TestMaintenanceState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11154 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839641/HDFS-11154-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Updated] (HDFS-10899) Add functionality to re-encrypt EDEKs.
[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-10899: - Attachment: HDFS-10899.wip.2.patch > Add functionality to re-encrypt EDEKs. > -- > > Key: HDFS-10899 > URL: https://issues.apache.org/jira/browse/HDFS-10899 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-10899.wip.2.patch, HDFS-10899.wip.patch, Re-encrypt > edek design doc.pdf > > > Currently when an encryption zone (EZ) key is rotated, it only takes effect > on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key > rotation, for improved security. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11148) Update DataNode to use StorageLocationChecker at startup
[
https://issues.apache.org/jira/browse/HDFS-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678348#comment-15678348
]
Hadoop QA commented on HDFS-11148:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 285 unchanged - 1 fixed = 289 total (was 286) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 2s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11148 |
| GITHUB PR | https://github.com/apache/hadoop/pull/162 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c8847131fb2d 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7584fbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17611/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17611/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17611/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Commented] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678337#comment-15678337
]
Hadoop QA commented on HDFS-10966:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 693 unchanged - 3 fixed = 699 total (was 696) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 61m
6s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10966 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839654/HDFS-10966.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8ba1d89a341b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7584fbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17613/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17613/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17613/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL:
[jira] [Updated] (HDFS-11159) Add reencryptEDEK interface for KMS
[ https://issues.apache.org/jira/browse/HDFS-11159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-11159: - Status: Patch Available (was: Open) > Add reencryptEDEK interface for KMS > --- > > Key: HDFS-11159 > URL: https://issues.apache.org/jira/browse/HDFS-11159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-11159.01.patch > > > This is the KMS part. Please refer to HDFS-10899 for the design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11159) Add reencryptEDEK interface for KMS
[ https://issues.apache.org/jira/browse/HDFS-11159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-11159: - Attachment: HDFS-11159.01.patch Patch 1 attached. Ideally this should be a hadoop jira but no such option as a subtask of HDFS-10899. > Add reencryptEDEK interface for KMS > --- > > Key: HDFS-11159 > URL: https://issues.apache.org/jira/browse/HDFS-11159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: encryption, kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-11159.01.patch > > > This is the KMS part. Please refer to HDFS-10899 for the design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11159) Add reencryptEDEK interface for KMS
Xiao Chen created HDFS-11159: Summary: Add reencryptEDEK interface for KMS Key: HDFS-11159 URL: https://issues.apache.org/jira/browse/HDFS-11159 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption, kms Reporter: Xiao Chen Assignee: Xiao Chen This is the KMS part. Please refer to HDFS-10899 for the design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8824) Do not use small blocks for balancing the cluster
[ https://issues.apache.org/jira/browse/HDFS-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678217#comment-15678217 ] Zhe Zhang commented on HDFS-8824: - Good finding Kihwal. +1 on extending the documentation with this effect. Should we change the default value of 10MB? > Do not use small blocks for balancing the cluster > - > > Key: HDFS-8824 > URL: https://issues.apache.org/jira/browse/HDFS-8824 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: h8824_20150727b.patch, h8824_20150811b.patch > > > Balancer gets datanode block lists from NN and then move the blocks in order > to balance the cluster. It should not use the blocks with small size since > moving the small blocks generates a lot of overhead and the small blocks do > not help balancing the cluster much. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678192#comment-15678192
]
Zhe Zhang edited comment on HDFS-10966 at 11/19/16 12:22 AM:
-
Thanks Kihwal for the review! Uploading a new patch:
# This patch does change the Balancer behavior introduced by HDFS-4261, around
the timeout logic. But I don't think there's a negative effect. By staying in
the {{dispatchBlocks}} while loop longer, the only overhead is to check
{{chooseNextMove}}, which only checks local states, without issuing NameNode
workload. Even if we jump out of the while loop, the thread for that Source
cannot be reused at another Source anyway. In {{TestBalancer}} I reset the
config value to 5s, and the run time is normal.
# Added to {{hdfs-default.xml}}, thx for the catch.
# I think it is a good idea, added.
was (Author: zhz):
Thanks Kihwal for the review! Uploading a new patch:
# This patch does change the Balancer behavior introduced by HDFS-4261, around
the timeout logic. But I don't think there's a negative effect. By staying in
the {{dispatchBlocks}} while loop longer, the only overhead is to check
{{chooseNextMove}}, which only checks local states, without issuing NameNode
workload. Even if we jump out of the while loop, the thread for that Source
cannot be reused at another Source anyway.
# Added to {{hdfs-default.xml}}, thx for the catch.
# I think it is a good idea, added.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch, HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10966:
-
Attachment: HDFS-10966.01.patch
Thanks Kihwal for the review! Uploading a new patch:
# This patch does change the Balancer behavior introduced by HDFS-4261, around
the timeout logic. But I don't think there's a negative effect. By staying in
the {{dispatchBlocks}} while loop longer, the only overhead is to check
{{chooseNextMove}}, which only checks local states, without issuing NameNode
workload. Even if we jump out of the while loop, the thread for that Source
cannot be reused at another Source anyway.
# Added to {{hdfs-default.xml}}, thx for the catch.
# I think it is a good idea, added.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch, HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11138: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~vagarychen] Thank you for contributing this patch. I have committed this to ozone branch. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: HDFS-7240 > > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11154: Status: Patch Available (was: Open) > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11158) Ozone: TestDatanodeStateMachine is failing intermittently
Anu Engineer created HDFS-11158: --- Summary: Ozone: TestDatanodeStateMachine is failing intermittently Key: HDFS-11158 URL: https://issues.apache.org/jira/browse/HDFS-11158 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Anu Engineer Assignee: Anu Engineer Failed in HDFS-11138 patch v6 run -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11148) Update DataNode to use StorageLocationChecker at startup
[
https://issues.apache.org/jira/browse/HDFS-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15678120#comment-15678120
]
Hadoop QA commented on HDFS-11148:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 285 unchanged - 1 fixed = 288 total (was 286) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11148 |
| GITHUB PR | https://github.com/apache/hadoop/pull/162 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux faaa6181bf71 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7584fbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17610/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17610/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17610/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.002.patch > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated HDFS-11156:
-
Target Version/s: 3.0.0-alpha2 (was: 2.8.0)
+1 for the idea.
I change the target version as 3.0+. Hope you don't mind.
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677917#comment-15677917
]
Lei (Eddy) Xu commented on HDFS-9668:
-
Hi, [~arpiagariu] Thanks for raising the concerns. I believe that we would
agree that [~jingcheng...@intel.com] has done a great work here.
A suggestion to [~jingcheng...@intel.com]. Would you mind to write a short
summary / design doc to highlight the lock policy you proposed. It'd would be
very helpful for us to reason from the big picture in design, and uses the
guidance to understand the actual implementation: so that we could understand
which part of code is by design and which part is bug. I understand that for so
many iterations, it would be frustrated and confusing to address so many
aspects of comments.
Really appreciated that you kindly answer all of the comments raised here.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-10.patch,
> HDFS-9668-11.patch, HDFS-9668-12.patch, HDFS-9668-13.patch,
> HDFS-9668-14.patch, HDFS-9668-14.patch, HDFS-9668-15.patch,
> HDFS-9668-16.patch, HDFS-9668-17.patch, HDFS-9668-18.patch,
> HDFS-9668-19.patch, HDFS-9668-19.patch, HDFS-9668-2.patch,
> HDFS-9668-20.patch, HDFS-9668-21.patch, HDFS-9668-22.patch,
> HDFS-9668-23.patch, HDFS-9668-23.patch, HDFS-9668-3.patch, HDFS-9668-4.patch,
> HDFS-9668-5.patch, HDFS-9668-6.patch, HDFS-9668-7.patch, HDFS-9668-8.patch,
> HDFS-9668-9.patch, execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
>
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677911#comment-15677911
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 89m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839610/HDFS-11138-HDFS-7240.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b553fc17dd47 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 5ceca1b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17609/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically
[jira] [Comment Edited] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677871#comment-15677871 ] Anu Engineer edited comment on HDFS-11138 at 11/18/16 9:53 PM: --- +1, Thanks for updating this patch. I will commit this patch to ozone branch after Jenkins. was (Author: anu): +1, Thanks for updating this patch. I will commit this patch to ozone branch shortly. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677871#comment-15677871 ] Anu Engineer commented on HDFS-11138: - +1, Thanks for updating this patch. I will commit this patch to ozone branch shortly. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11133) Ozone: Add allocateContainer RPC
[ https://issues.apache.org/jira/browse/HDFS-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11133: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~vagarychen], [~jnp] Thanks for the reviews. I have committed this to the feature branch. > Ozone: Add allocateContainer RPC > > > Key: HDFS-11133 > URL: https://issues.apache.org/jira/browse/HDFS-11133 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: oz >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11133-HDFS-7240.001.patch > > > Add allocateContainer RPC in SCM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11138) Block Storage : add block storage server
[ https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11138: -- Attachment: HDFS-11138-HDFS-7240.006.patch v006 patch to resolve the checkstyle warnings. > Block Storage : add block storage server > > > Key: HDFS-11138 > URL: https://issues.apache.org/jira/browse/HDFS-11138 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11138-HDFS-7240.001.patch, > HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch, > HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch, > HDFS-11138-HDFS-7240.006.patch > > > This JIRA adds the skeleton for server side code, which is one of the core > components in block storage. For performance concerns, the server does not > handle any actual read/write operation but serving primarily as a meta data > server. It provides four APIs: > # create volume : which will call into underlying container layer to allocate > containers > # delete volume : delete a specific volume (as well as its containers) > # info volume : return information of a specific volume > # list volume : list all volumes > Note that this is still subject to potentially major changes. Features such > as persistence are missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677614#comment-15677614
]
Hadoop QA commented on HDFS-11138:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
3s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 9
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11138 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839597/HDFS-11138-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 78f71dbd9b2b 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 5ceca1b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17608/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[
https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677554#comment-15677554
]
Hadoop QA commented on HDFS-11094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 9 new + 252 unchanged - 12 fixed = 261 total (was 264) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11094 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839584/HDFS-11094.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 2e86695a51a5 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f6ffa11 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17607/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17607/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17607/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17607/console |
| Powered by | Apache Yetus
[jira] [Commented] (HDFS-8824) Do not use small blocks for balancing the cluster
[ https://issues.apache.org/jira/browse/HDFS-8824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677540#comment-15677540 ] Kihwal Lee commented on HDFS-8824: -- While this will initially increase the efficiency of balancing, it is not without a negative side-effect. Older nodes in a cluster will slowly filled with smaller blocks as time goes on. This is accelerated if the cluster is heterogeneous. The smaller nodes will fill up more quickly/frequently and the balancer will move only big blocks out of those nodes. As more balacing happens, those nodes will contain more and more small blocks. If sufficient time passes, the blocks on those nodes will almost entirely small. This feature can be enabled for quickly resolving a storage balance issue, but long-term use can have unintended side-effect. Fortunately, we have not released any (other than alpha) with this feature. We can include more information in the release note and/or address the issue in the code/config. > Do not use small blocks for balancing the cluster > - > > Key: HDFS-8824 > URL: https://issues.apache.org/jira/browse/HDFS-8824 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: h8824_20150727b.patch, h8824_20150811b.patch > > > Balancer gets datanode block lists from NN and then move the blocks in order > to balance the cluster. It should not use the blocks with small size since > moving the small blocks generates a lot of overhead and the small blocks do > not help balancing the cluster much. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677513#comment-15677513
]
Arpit Agarwal commented on HDFS-9668:
-
bq. Shall we get some consensus of the purposes for each lock (read/write locks
and ReplicaMap#mutex) before we commit this ?
Yes. The proposed locking semantics should be clear before we start reviewing.
I am -1 on the current changes.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-10.patch,
> HDFS-9668-11.patch, HDFS-9668-12.patch, HDFS-9668-13.patch,
> HDFS-9668-14.patch, HDFS-9668-14.patch, HDFS-9668-15.patch,
> HDFS-9668-16.patch, HDFS-9668-17.patch, HDFS-9668-18.patch,
> HDFS-9668-19.patch, HDFS-9668-19.patch, HDFS-9668-2.patch,
> HDFS-9668-20.patch, HDFS-9668-21.patch, HDFS-9668-22.patch,
> HDFS-9668-23.patch, HDFS-9668-23.patch, HDFS-9668-3.patch, HDFS-9668-4.patch,
> HDFS-9668-5.patch, HDFS-9668-6.patch, HDFS-9668-7.patch, HDFS-9668-8.patch,
> HDFS-9668-9.patch, execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can
[jira] [Commented] (HDFS-11157) Enhance documentation around cpLock
[
https://issues.apache.org/jira/browse/HDFS-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677451#comment-15677451
]
Zhe Zhang commented on HDFS-11157:
--
Pinging [~kihwal] [~andrew.wang] [~atm] to share some thoughts.
> Enhance documentation around cpLock
> ---
>
> Key: HDFS-11157
> URL: https://issues.apache.org/jira/browse/HDFS-11157
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation, ha, namenode
>Reporter: Zhe Zhang
>Priority: Minor
>
> The {{cpLock}} was introduced in HDFS-7097. Among all operations allowed on
> SbNN, some acquire this lock (e.g. {{restoreFailedStorage}}) and some do not,
> including {{metasave}}, {{refreshNodes}}, {{setSafeMode}}.
> We should enhance the documentation around {{cpLock}} to explain the above.
> Also, maybe {{metasave}} and {{refreshNodes}} do not need fsn write lock?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11157) Enhance documentation around cpLock
Zhe Zhang created HDFS-11157:
Summary: Enhance documentation around cpLock
Key: HDFS-11157
URL: https://issues.apache.org/jira/browse/HDFS-11157
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation, ha, namenode
Reporter: Zhe Zhang
Priority: Minor
The {{cpLock}} was introduced in HDFS-7097. Among all operations allowed on
SbNN, some acquire this lock (e.g. {{restoreFailedStorage}}) and some do not,
including {{metasave}}, {{refreshNodes}}, {{setSafeMode}}.
We should enhance the documentation around {{cpLock}} to explain the above.
Also, maybe {{metasave}} and {{refreshNodes}} do not need fsn write lock?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11133) Ozone: Add allocateContainer RPC
[ https://issues.apache.org/jira/browse/HDFS-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677418#comment-15677418 ] Anu Engineer commented on HDFS-11133: - [~jnp] I will fix that constructor issue while committing, thanks for the review. > Ozone: Add allocateContainer RPC > > > Key: HDFS-11133 > URL: https://issues.apache.org/jira/browse/HDFS-11133 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: oz >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11133-HDFS-7240.001.patch > > > Add allocateContainer RPC in SCM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11138) Block Storage : add block storage server
[
https://issues.apache.org/jira/browse/HDFS-11138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chen Liang updated HDFS-11138:
--
Attachment: HDFS-11138-HDFS-7240.005.patch
Upload v005 patch to address [~anu]'s comments.
Fixing comments 3, 4 and 6 rely on changing the data structure in
{{VolumeDescriptor}}, which, as discussed, will be revisited later. The rest
are fixed in v005 patch.
> Block Storage : add block storage server
>
>
> Key: HDFS-11138
> URL: https://issues.apache.org/jira/browse/HDFS-11138
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-11138-HDFS-7240.001.patch,
> HDFS-11138-HDFS-7240.002.patch, HDFS-11138-HDFS-7240.003.patch,
> HDFS-11138-HDFS-7240.004.patch, HDFS-11138-HDFS-7240.005.patch
>
>
> This JIRA adds the skeleton for server side code, which is one of the core
> components in block storage. For performance concerns, the server does not
> handle any actual read/write operation but serving primarily as a meta data
> server. It provides four APIs:
> # create volume : which will call into underlying container layer to allocate
> containers
> # delete volume : delete a specific volume (as well as its containers)
> # info volume : return information of a specific volume
> # list volume : list all volumes
> Note that this is still subject to potentially major changes. Features such
> as persistence are missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11126) Ozone: Add small file support RPC
[ https://issues.apache.org/jira/browse/HDFS-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11126: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~vagarychen], [~xyao], [~jnp] Thanks for the code review and comments. I have committed this to the feature branch. > Ozone: Add small file support RPC > - > > Key: HDFS-11126 > URL: https://issues.apache.org/jira/browse/HDFS-11126 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11126-HDFS-7240.001.patch, > HDFS-11126-HDFS-7240.002.patch > > > Add an RPC to send both data and metadata together in one RPC. This is useful > when we want to read and write small files, say less than 1 MB. This API is > very useful for ozone and cBlocks (HDFS-8) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11090) Leave safemode immediately if all blocks have reported in
[ https://issues.apache.org/jira/browse/HDFS-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677300#comment-15677300 ] Konstantin Shvachko commented on HDFS-11090: Andrew, if you target empty cluster case here, you should probably update the title/description of the jira and define what exactly "empty" means. > Leave safemode immediately if all blocks have reported in > - > > Key: HDFS-11090 > URL: https://issues.apache.org/jira/browse/HDFS-11090 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.7.3 >Reporter: Andrew Wang >Assignee: Yiqun Lin > Attachments: HDFS-11090.001.patch > > > Startup safemode is triggered by two thresholds: % blocks reported in, and > min # datanodes. It's extended by an interval (default 30s) until these two > thresholds are met. > Safemode extension is helpful when the cluster has data, and the default % > blocks threshold (0.99) is used. It gives DNs a little extra time to report > in and thus avoid unnecessary replication work. > However, we can leave startup safemode early if 100% of blocks have reported > in. > Note that operators sometimes change the % blocks threshold to > 1 to never > automatically leave safemode. We should maintain this behavior. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10941) Improve BlockManager#processMisReplicatesAsync log
[
https://issues.apache.org/jira/browse/HDFS-10941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10941:
-
Fix Version/s: 2.7.4
Thanks for the work Chen. I just backported to branch-2.7.
> Improve BlockManager#processMisReplicatesAsync log
> --
>
> Key: HDFS-10941
> URL: https://issues.apache.org/jira/browse/HDFS-10941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Xiaoyu Yao
>Assignee: Chen Liang
> Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10941.001.patch, HDFS-10941.002.patch,
> HDFS-10941.002.patch, HDFS-10941.003.patch
>
>
> BlockManager#processMisReplicatesAsync is the daemon thread running inside
> namenode to handle miserplicated blocks. As shown below, it has a trace log
> for each of the block in the cluster being processed (1 blocks per
> iteration after sleep 10s).
> {code}
> MisReplicationResult res = processMisReplicatedBlock(block);
> if (LOG.isTraceEnabled()) {
> LOG.trace("block " + block + ": " + res);
> }
> {code}
> However, it is not very useful as dumping every block in the cluster will
> overwhelm the namenode log without much useful information assuming the
> majority of the blocks are not over/under replicated. This ticket is opened
> to improve the log for easy troubleshooting of block replication related
> issues by:
>
> 1) add debug log for blocks that get under/over replicated result during
> {{processMisReplicatedBlock()}}
> 2) or change to trace log for only blocks that get non-OK result during
> {{processMisReplicatedBlock()}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11133) Ozone: Add allocateContainer RPC
[ https://issues.apache.org/jira/browse/HDFS-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677202#comment-15677202 ] Jitendra Nath Pandey edited comment on HDFS-11133 at 11/18/16 5:44 PM: --- In the new constructor for LevelDBStore there is a bit of code duplication with the existing constructor. It might make sense to define a constructor that takes 'Option' as an argument. +1 otherwise. was (Author: jnp): In the new constructor for LevelDBStore there is a bit of code duplication with the existing constructor. It might make sense to define a constructor that takes 'Option' as an argument. > Ozone: Add allocateContainer RPC > > > Key: HDFS-11133 > URL: https://issues.apache.org/jira/browse/HDFS-11133 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: oz >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11133-HDFS-7240.001.patch > > > Add allocateContainer RPC in SCM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11133) Ozone: Add allocateContainer RPC
[ https://issues.apache.org/jira/browse/HDFS-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677202#comment-15677202 ] Jitendra Nath Pandey commented on HDFS-11133: - In the new constructor for LevelDBStore there is a bit of code duplication with the existing constructor. It might make sense to define a constructor that takes 'Option' as an argument. > Ozone: Add allocateContainer RPC > > > Key: HDFS-11133 > URL: https://issues.apache.org/jira/browse/HDFS-11133 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: oz >Reporter: Anu Engineer >Assignee: Anu Engineer > Fix For: HDFS-7240 > > Attachments: HDFS-11133-HDFS-7240.001.patch > > > Add allocateContainer RPC in SCM. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HDFS-11094: --- Attachment: HDFS-11094.007.patch I'm addressing the logging and checkstyle warnings in this patch. However, I will need some time to figure out how to do the unit testing. [~liuml07], do you have any suggestions? It seems like this will be quite difficult to mock out. > Send back HAState along with NamespaceInfo during a versionRequest as an > optional parameter > --- > > Key: HDFS-11094 > URL: https://issues.apache.org/jira/browse/HDFS-11094 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, > HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, > HDFS-11094.006.patch, HDFS-11094.007.patch > > > The datanode should know which NN is active when it is connecting/registering > to the NN. Currently, it only figures this out during its first (and > subsequent) heartbeat(s) and so there is a period of time where the datanode > is alive and registered, but can't actually do anything because it doesn't > know which NN is active. A byproduct of this is that the MiniDFSCluster will > become active before it knows what NN is active, which can lead to NPEs when > calling getActiveNN(). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11112) Journal Nodes should refuse to format non-empty directories
[
https://issues.apache.org/jira/browse/HDFS-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677124#comment-15677124
]
Wei-Chiu Chuang commented on HDFS-2:
Hi [~arpitagarwal]
a while back I also studied the behavior of JN in this situation (with minor
deviation) after finished HDFS-10360 to deal with the similar issue in
DataNodes. Here's what I saw. Could you comment if this is also what you saw?
Maybe the situation is different, and from what I saw JN didn't automatically
format the directory.
{quote}
if current/VERSION is missing, it will start with no problem like nothing has
happened. No auto-format. But when NN tries to send edits to the JN, the JN
will respond with JournalNotFormattedException. Cloudera Manager dashboard will
see JN and NN in bad health.
2016-06-07 11:14:18,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3
on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournal
Protocol.getJournalState from 172.26.10.80:36663 Call#7 Retry#0
org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal
Storage Directory /data/jn/nameservice1 not formatted
at
org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:472)
at
org.apache.hadoop.hdfs.qjournal.server.Journal.getLastPromisedEpoch(Journal.java:245)
at
org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getJournalState(JournalNodeRpcServer.java:124)
at
org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getJournalState(QJournalProtocolServerSideTranslat
orPB.java:118)
at
org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.
java:25415)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
If I remove current/VERSION from all three JNs,
NN will shutdown because it's unabel to get a quorum.
If I put the file back to an active JN, it will still complain about it. But if
I restart the JN with the file in place, the JN will function normally.
So in the case of JN, there is no permanent harm and the damage can be reverted.
{quote}
[~linyiqun] I think the patch looks good to me overall. Do you know the
behavior of JN after your patch? Does JN simply ignores format? Does JN
continues to function normally?
> Journal Nodes should refuse to format non-empty directories
> ---
>
> Key: HDFS-2
> URL: https://issues.apache.org/jira/browse/HDFS-2
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Yiqun Lin
> Attachments: HDFS-2.001.patch, HDFS-2.002.patch
>
>
> Journal Nodes should reject the {{format}} RPC request if a storage directory
> is non-empty. The relevant code is in {{JNStorage#format}}.
> {code}
> void format(NamespaceInfo nsInfo) throws IOException {
> setStorageInfo(nsInfo);
> ...
> unlockAll();
> sd.clearDirectory();
> writeProperties(sd);
> createPaxosDir();
> analyzeStorage();
> {code}
> This would make the behavior similar to {{namenode -format -nonInteractive}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-303) Make contracts of LocalFileSystem and DistributedFileSystem consistent
[
https://issues.apache.org/jira/browse/HDFS-303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677026#comment-15677026
]
Andras Bokor commented on HDFS-303:
---
bq. When trying to make a directory under an existing file, HDFS throws an
IOException while LocalFileSystem doesn't.
HADOOP-6229 solves this
bq. The FileSytem#listStatus(Path) method returns null for a non-existent file
on HDFS, while LocalFileSytem returns an empty FileStatus array.
In HADOOP-6201 and HDFS-538 it was agreed that FileSystem::listStatus should
throw FileNotFoundException instead of returning null, when the target
directory does not exist.
In addition in {{DistributedFilesystem}}:
{code:title=DistributedFilesystem.java:862}
if (thisListing == null) { // the directory does not exist
throw new FileNotFoundException("File " + p + " does not exist.");
}
{code}
In {{RawLocalFileSystem}}:
{code}
if (!localf.exists()) {
throw new FileNotFoundException("File " + f + " does not exist");
}
{code}
So it seems HDFS throws exception too.
bq. When trying to rename a non-existent path, LocalFileSystem throws an
IOException, while HDFS returns false.
HDFS-6262. I wrote a comment that I think that ticket should be closed as
"Won't fix".
bq. When renaming a file or directory to a non-existent directory (e.g. /a/b to
/c/d, where /c doesn't exist) LocalFileSystem succeeds (returns true) while
HDFS fails (false).
I think it is the same as was discussed in HADOOP-13082
bq. When renaming a file (or directory) as an existing file (or directory)
LocalFileSystem succeeds (returns true) while HDFS fails (false).
The discussion in HADOOP-13082 covers this as well.
Based on this I think the ticket can be closed. Do you have any objection?
> Make contracts of LocalFileSystem and DistributedFileSystem consistent
> --
>
> Key: HDFS-303
> URL: https://issues.apache.org/jira/browse/HDFS-303
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tom White
>Assignee: Andras Bokor
> Attachments: HDFS-303-common-test-case.patch, HDFS-303.patch,
> hadoop-4114.patch
>
>
> There are a number of edge cases that the two file system implementations
> handle differently. In particular:
> * When trying to make a directory under an existing file, HDFS throws an
> IOException while LocalFileSystem doesn't.
> * The FileSytem#listStatus(Path) method returns null for a non-existent file
> on HDFS, while LocalFileSytem returns an empty FileStatus array.
> * When trying to rename a non-existent path, LocalFileSystem throws an
> IOException, while HDFS returns false.
> * When renaming a file or directory to a non-existent directory (e.g. /a/b to
> /c/d, where /c doesn't exist) LocalFileSystem succeeds (returns true) while
> HDFS fails (false).
> * When renaming a file (or directory) as an existing file (or directory)
> LocalFileSystem succeeds (returns true) while HDFS fails (false).
> We should document the expected behaviour for these cases in FileSystem's
> javadoc, and make sure all implementations conform to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11155) VolumeScanner should report the latest generation stamp of a bad replica
[
https://issues.apache.org/jira/browse/HDFS-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-11155:
---
Description:
HDFS-10512 fixed a race condition that caused VolumeScanner to terminate
abruptly when a corrupt replica, which is being updated, is detected. However,
when such a corrupt replica is detected, VolumeScanner still reports the old
replica generation stamp to the NN. NN then directs DN to remove the older
replica. Because the generation stamp is updated, DN can not find it, so
corrupt replica remains corrupt.
NN's log shows something similar to the following:
{quote}
2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK
NameSystem.addToCorruptReplicasMap: blk_1077571736 added as corrupt on
192.168.168.58:50010 by /192.168.168.58 because client machine reported it
2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK* invalidateBlock:
blk_1077571736_3991953(stored=blk_1077571736_3992018) on 192.168.168.58:50010
{quote}
The DN's log has these:
{noformat}
2016-11-17 21:08:04,815 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Appending
to FinalizedReplica, blk_1077571736_3991953, FINALIZED
getNumBytes() = 39061752
getBytesOnDisk() = 39061752
getVisibleLength()= 39061752
getVolume() = /data/3/dfs/dn/current
getBlockFile()=
/data/3/dfs/dn/current/BP-1092022411-192.168.168.55-1474407949037/current/finalized/subdir58/subdir112/blk_1077571736
2016-11-17 21:08:09,158 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed to
delete replica blk_1077571736_3991953: ReplicaInfo not found.
{noformat}
was:
HDFS-10512 fixed a race condition that caused VolumeScanner to terminate
abruptly when a corrupt replica is detected. However, when a corrupt replica is
detected, VolumeScanner still reports the old replica generation stamp to the
NN. NN then directs DN to remove the older replica, but because the generation
stamp is updated, DN can not find it, so corrupt replica remains corrupt.
NN's log shows something similar to the following:
{quote}
2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK
NameSystem.addToCorruptReplicasMap: blk_1077571736 added as corrupt on
192.168.168.58:50010 by /192.168.168.58 because client machine reported it
2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK* invalidateBlock:
blk_1077571736_3991953(stored=blk_1077571736_3992018) on 192.168.168.58:50010
{quote}
The DN's log has these:
{noformat}
2016-11-17 21:08:04,815 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Appending
to FinalizedReplica, blk_1077571736_3991953, FINALIZED
getNumBytes() = 39061752
getBytesOnDisk() = 39061752
getVisibleLength()= 39061752
getVolume() = /data/3/dfs/dn/current
getBlockFile()=
/data/3/dfs/dn/current/BP-1092022411-192.168.168.55-1474407949037/current/finalized/subdir58/subdir112/blk_1077571736
2016-11-17 21:08:09,158 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed to
delete replica blk_1077571736_3991953: ReplicaInfo not found.
{noformat}
> VolumeScanner should report the latest generation stamp of a bad replica
>
>
> Key: HDFS-11155
> URL: https://issues.apache.org/jira/browse/HDFS-11155
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.4
> Environment: CDH5.7.3
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> HDFS-10512 fixed a race condition that caused VolumeScanner to terminate
> abruptly when a corrupt replica, which is being updated, is detected.
> However, when such a corrupt replica is detected, VolumeScanner still reports
> the old replica generation stamp to the NN. NN then directs DN to remove the
> older replica. Because the generation stamp is updated, DN can not find it,
> so corrupt replica remains corrupt.
> NN's log shows something similar to the following:
> {quote}
> 2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1077571736 added as corrupt on
> 192.168.168.58:50010 by /192.168.168.58 because client machine reported it
> 2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK* invalidateBlock:
> blk_1077571736_3991953(stored=blk_1077571736_3992018) on 192.168.168.58:50010
> {quote}
> The DN's log has these:
> {noformat}
> 2016-11-17 21:08:04,815 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
> Appending to FinalizedReplica, blk_1077571736_3991953, FINALIZED
> getNumBytes() = 39061752
> getBytesOnDisk() = 39061752
> getVisibleLength()= 39061752
> getVolume() = /data/3/dfs/dn/current
> getBlockFile()=
>
[jira] [Updated] (HDFS-11155) VolumeScanner should report the latest generation stamp of a bad replica
[
https://issues.apache.org/jira/browse/HDFS-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-11155:
---
Environment: CDH5.7.3 (was: CDH5.7.2)
> VolumeScanner should report the latest generation stamp of a bad replica
>
>
> Key: HDFS-11155
> URL: https://issues.apache.org/jira/browse/HDFS-11155
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.4
> Environment: CDH5.7.3
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>
> HDFS-10512 fixed a race condition that caused VolumeScanner to terminate
> abruptly when a corrupt replica is detected. However, when a corrupt replica
> is detected, VolumeScanner still reports the old replica generation stamp to
> the NN. NN then directs DN to remove the older replica, but because the
> generation stamp is updated, DN can not find it, so corrupt replica remains
> corrupt.
> NN's log shows something similar to the following:
> {quote}
> 2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1077571736 added as corrupt on
> 192.168.168.58:50010 by /192.168.168.58 because client machine reported it
> 2016-11-17 21:08:05,350 INFO BlockStateChange: BLOCK* invalidateBlock:
> blk_1077571736_3991953(stored=blk_1077571736_3992018) on 192.168.168.58:50010
> {quote}
> The DN's log has these:
> {noformat}
> 2016-11-17 21:08:04,815 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
> Appending to FinalizedReplica, blk_1077571736_3991953, FINALIZED
> getNumBytes() = 39061752
> getBytesOnDisk() = 39061752
> getVisibleLength()= 39061752
> getVolume() = /data/3/dfs/dn/current
> getBlockFile()=
> /data/3/dfs/dn/current/BP-1092022411-192.168.168.55-1474407949037/current/finalized/subdir58/subdir112/blk_1077571736
> 2016-11-17 21:08:09,158 INFO
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Failed
> to delete replica blk_1077571736_3991953: ReplicaInfo not found.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11156:
---
Summary: Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with
FileSystem API (was: WebHDFS GET_BLOCK_LOCATIONS )
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-11156) WebHDFS GET_BLOCK_LOCATIONS
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HDFS-11156 started by Weiwei Yang.
--
> WebHDFS GET_BLOCK_LOCATIONS
>
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11156) WebHDFS GET_BLOCK_LOCATIONS
Weiwei Yang created HDFS-11156:
--
Summary: WebHDFS GET_BLOCK_LOCATIONS
Key: HDFS-11156
URL: https://issues.apache.org/jira/browse/HDFS-11156
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.7.3
Reporter: Weiwei Yang
Assignee: Weiwei Yang
Following webhdfs REST API
{code}
http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
{code}
will get a response like
{code}
{
"LocatedBlocks" : {
"fileLength" : 1073741824,
"isLastBlockComplete" : true,
"isUnderConstruction" : false,
"lastLocatedBlock" : { ... },
"locatedBlocks" : [ {...} ]
}
}
{code}
This represents for *o.a.h.h.p.LocatedBlocks*. However according to
*FileSystem* API,
{code}
public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
{code}
clients would expect an array of BlockLocation. This mismatch should be fixed.
Marked as Incompatible change as this will change the output of the
GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11151) [SPS]: StoragePolicySatisfier should gracefully handle when there is no target node with the required storage type
[ https://issues.apache.org/jira/browse/HDFS-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-11151: Summary: [SPS]: StoragePolicySatisfier should gracefully handle when there is no target node with the required storage type (was: [SPS]: Handle unable to choose target node for the required storage type by StoragePolicySatisfier) > [SPS]: StoragePolicySatisfier should gracefully handle when there is no > target node with the required storage type > -- > > Key: HDFS-11151 > URL: https://issues.apache.org/jira/browse/HDFS-11151 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-11151-HDFS-10285.00.patch > > > Presently SPS is not handling a case where the failed to choose target node > for the required storage type. In general, there are two cases: > # For the given path, unable to find any target node for any of its blocks or > block locations(src nodes). Means, no block movement will be scheduled > against this path. > # For the given path, there are few target nodes available for few block > locations(source nodes). Means, some of the blocks or block locations(src > nodes) under the given path will be scheduled for block movement. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11117) Refactor striped file tests to allow flexibly test erasure coding policy
[ https://issues.apache.org/jira/browse/HDFS-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676429#comment-15676429 ] Hudson commented on HDFS-7: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10862 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10862/]) HDFS-7. Refactor striped file tests to allow flexibly test erasure (kai.zheng: rev f6ffa11635c47030a91d420da942da1fb425eb49) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure090.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure020.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure070.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure120.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileChecksum.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure170.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReconstructStripedBlocksWithRackAwareness.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure030.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure060.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure160.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithMissingBlocks.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSequentialBlockGroupId.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure100.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure000.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure210.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/mover/TestMover.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/util/TestStripedBlockUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/StripedFileTestUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure080.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommissionWithStriped.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicyWithSnapshot.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure190.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecoveryStriped.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReadStripedFileWithDecoding.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSortLocatedStripedBlock.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure110.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure010.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStreamWithFailure150.java * (edit)
[jira] [Commented] (HDFS-11111) Delete something in .Trash using "rm" should be forbidden without safety option
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676393#comment-15676393
]
Hadoop QA commented on HDFS-1:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 11s{color}
| {color:red} root generated 1 new + 692 unchanged - 0 fixed = 693 total (was
692) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 35 new + 9 unchanged - 0 fixed = 44 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 38s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestTrash |
| | hadoop.cli.TestCLI |
| | hadoop.fs.viewfs.TestViewFsTrash |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-1 |
| GITHUB PR | https://github.com/apache/hadoop/pull/159 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 14f8a115d272 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c0b1a44 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/17606/artifact/patchprocess/diff-compile-javac-root.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17606/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/17606/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17606/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17606/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
[jira] [Updated] (HDFS-11117) Refactor striped file tests to allow flexibly test erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kai Zheng updated HDFS-7:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Sammi for the contribution!
> Refactor striped file tests to allow flexibly test erasure coding policy
>
>
> Key: HDFS-7
> URL: https://issues.apache.org/jira/browse/HDFS-7
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: SammiChen
>Assignee: SammiChen
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-7-v1.patch, HDFS-7-v2.patch,
> HDFS-7-v3.patch, HDFS-7-v4.patch, HDFS-7-v5.patch
>
>
> This task is going to refactor current striped file test case structures,
> especially {{StripedFileTestUtil}} file which is used in many striped file
> test cases. All current striped file test cases only support one erasure
> coding policy, that's the default RS-DEFAULT-6-3-64k policy. The goal of the
> refactor is to make the structures more convenient to support other erasure
> coding policies, such as XOR policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11117) Refactor striped file tests to allow flexibly test erasure coding policy
[
https://issues.apache.org/jira/browse/HDFS-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676372#comment-15676372
]
Kai Zheng commented on HDFS-7:
--
Thanks [~Sammi] for the hard working and great refactoring. It will make it
much easier to support other policy testing. +1 on latest patch.
> Refactor striped file tests to allow flexibly test erasure coding policy
>
>
> Key: HDFS-7
> URL: https://issues.apache.org/jira/browse/HDFS-7
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: SammiChen
>Assignee: SammiChen
> Attachments: HDFS-7-v1.patch, HDFS-7-v2.patch,
> HDFS-7-v3.patch, HDFS-7-v4.patch, HDFS-7-v5.patch
>
>
> This task is going to refactor current striped file test case structures,
> especially {{StripedFileTestUtil}} file which is used in many striped file
> test cases. All current striped file test cases only support one erasure
> coding policy, that's the default RS-DEFAULT-6-3-64k policy. The goal of the
> refactor is to make the structures more convenient to support other erasure
> coding policies, such as XOR policy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10206) getBlockLocations might not sort datanodes properly by distance
[
https://issues.apache.org/jira/browse/HDFS-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676336#comment-15676336
]
Nandakumar commented on HDFS-10206:
---
bq. From the below code, it seems each level will increase by 2.
Yes, you're right. Distance from a node to it's parent is assumed to be 1, so
adding the value of both the nodes will make it 2.
Sorry for the confusion on that.
{quote}
one is the reader being a datanode in a remote rack in a large cluster; for
that NetworkTopology already has the reader in its tree, it will be faster to
compare parents reference.
{quote}
We can use {{NetworkTopology.contains(node)}} to check if the reader is a
datanode and use {{NetworkTopology.getDistance(node1, node2)}} to get the
distance (which also calculates the distance by summing up the nodes distances
to their closest common ancestor), but both of these methods use
{{ReadWriteLock.readLock()}} which might again impact the performance.
Any comments on using {{NetworkTopology.contains(node)}} to check and use
{{NetworkTopology.getDistance(node1, node2)}} to get the distance in case if
the reader is an off rack datanode?
I'm currently working on the benchmarking, will update it once it's done.
> getBlockLocations might not sort datanodes properly by distance
> ---
>
> Key: HDFS-10206
> URL: https://issues.apache.org/jira/browse/HDFS-10206
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ming Ma
>Assignee: Nandakumar
> Attachments: HDFS-10206.000.patch
>
>
> If the DFSClient machine is not a datanode, but it shares its rack with some
> datanodes of the HDFS block requested, {{DatanodeManager#sortLocatedBlocks}}
> might not put the local-rack datanodes at the beginning of the sorted list.
> That is because the function didn't call {{networktopology.add(client);}} to
> properly set the node's parent node; something required by
> {{networktopology.sortByDistance}} to compute distance between two nodes in
> the same topology tree.
> Another issue with {{networktopology.sortByDistance}} is it only
> distinguishes local rack from remote rack, but it doesn't support general
> distance calculation to tell how remote the rack is.
> {noformat}
> NetworkTopology.java
> protected int getWeight(Node reader, Node node) {
> // 0 is local, 1 is same rack, 2 is off rack
> // Start off by initializing to off rack
> int weight = 2;
> if (reader != null) {
> if (reader.equals(node)) {
> weight = 0;
> } else if (isOnSameRack(reader, node)) {
> weight = 1;
> }
> }
> return weight;
> }
> {noformat}
> HDFS-10203 has suggested moving the sorting from namenode to DFSClient to
> address another issue. Regardless of where we do the sorting, we still need
> fix the issues outline here.
> Note that BlockPlacementPolicyDefault shares the same NetworkTopology object
> used by DatanodeManager and requires Nodes stored in the topology to be
> {{DatanodeDescriptor}} for block placement. So we need to make sure we don't
> pollute the NetworkTopology if we plan to fix it on the server side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11111) Delete something in .Trash using "rm" should be forbidden without safety option
[
https://issues.apache.org/jira/browse/HDFS-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lantao Jin updated HDFS-1:
--
Attachment: HDFS-1.1.patch
> Delete something in .Trash using "rm" should be forbidden without safety
> option
> -
>
> Key: HDFS-1
> URL: https://issues.apache.org/jira/browse/HDFS-1
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 2.7.2
>Reporter: Lantao Jin
> Labels: Incompatible
> Attachments: HDFS-1.1.patch
>
>
> As we discussed in HDFS-11102, double confirmation does not seem to be a
> graceful solution for users. Deleting files in .Trash accidentally is still
> an issue though. The behaviour of users I'm worried about is {{rm}} ing
> something in .Trash (without explicitly understanding that those files will
> not be recoverable). This is in contrast to {{rm}} ing something with
> "-skipTrash" option (That's a very purposeful action).
> So it is not the same case as HADOOP-12358. The solution is throwing an
> exception and remind user to add "-trash" option to delete dirs in trash for
> safely:
> {code}
> Can not delete somehing trash directly! Please add "-trash" or "-T" in "rm"
> command to do that.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11060) make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable
[ https://issues.apache.org/jira/browse/HDFS-11060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated HDFS-11060: -- Attachment: HDFS-11060.1.patch > make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable > - > > Key: HDFS-11060 > URL: https://issues.apache.org/jira/browse/HDFS-11060 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs >Reporter: Lantao Jin >Priority: Minor > Attachments: HDFS-11060.1.patch > > > Current, the easiest way to determine which blocks is missing is using NN web > UI or JMX. Unfortunately, because the > DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED=100 is hard code in FSNamesystem, > only 100 missing blocks can be return by UI and JMX. Even the result of URL > "https://nn:50070/fsck?listcorruptfileblocks=1=%2F; is limited by this > hard code value too. > I did know FSCK can return more than 100 result but due to the security > reason(with kerberos), it is very hard to involve to costumer programs and > scripts. > So I think it should add a configurable var "maxCorruptFileBlocksReturned" to > fix above case. > If community also think it's worth to do, I will patch this. If not, please > feel free to tell me the reason. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6874) Add GET_BLOCK_LOCATIONS operation to HttpFS
[
https://issues.apache.org/jira/browse/HDFS-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676204#comment-15676204
]
Hadoop QA commented on HDFS-6874:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-httpfs: The
patch generated 6 new + 423 unchanged - 1 fixed = 429 total (was 424) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-httpfs generated 3 new
+ 52 unchanged - 0 fixed = 55 total (was 52) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 30s{color}
| {color:red} hadoop-hdfs-httpfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.http.client.TestHttpFSFWithSWebhdfsFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-6874 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839524/HDFS-6874.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3827106ccca1 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c0b1a44 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17604/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/17604/artifact/patchprocess/diff-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17604/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17604/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U:
hadoop-hdfs-project/hadoop-hdfs-httpfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17604/console |
| Powered by
[jira] [Updated] (HDFS-6874) Add GET_BLOCK_LOCATIONS operation to HttpFS
[
https://issues.apache.org/jira/browse/HDFS-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-6874:
--
Status: Patch Available (was: In Progress)
> Add GET_BLOCK_LOCATIONS operation to HttpFS
> ---
>
> Key: HDFS-6874
> URL: https://issues.apache.org/jira/browse/HDFS-6874
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.3, 2.4.1
>Reporter: Gao Zhong Liang
>Assignee: Weiwei Yang
> Labels: BB2015-05-TBR
> Attachments: HDFS-6874-1.patch, HDFS-6874-branch-2.6.0.patch,
> HDFS-6874.02.patch, HDFS-6874.03.patch, HDFS-6874.patch
>
>
> GET_BLOCK_LOCATIONS operation is missing in HttpFS, which is already
> supported in WebHDFS. For the request of GETFILEBLOCKLOCATIONS in
> org.apache.hadoop.fs.http.server.HttpFSServer, BAD_REQUEST is returned so far:
> ...
> case GETFILEBLOCKLOCATIONS: {
> response = Response.status(Response.Status.BAD_REQUEST).build();
> break;
> }
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6874) Add GET_BLOCK_LOCATIONS operation to HttpFS
[
https://issues.apache.org/jira/browse/HDFS-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-6874:
--
Attachment: HDFS-6874.03.patch
Upload v3 patch that consolidates the changes from original patch in this
ticket. This patch has
- Add implementation to GET_BLOCK_LOCATIONS operation in httpfs
- Use GET_BLOCK_LOCATIONS as operation name in httpfs instead of original
GETFILEBLOCKLOCATIONS, to keep consistent with Webhdfs API
- Fixed an issue in TestHttpFSFWithWebhdfsFileSystem that it doesn't test
httpfs using webhdfs as proxy FS
> Add GET_BLOCK_LOCATIONS operation to HttpFS
> ---
>
> Key: HDFS-6874
> URL: https://issues.apache.org/jira/browse/HDFS-6874
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.4.1, 2.7.3
>Reporter: Gao Zhong Liang
>Assignee: Weiwei Yang
> Labels: BB2015-05-TBR
> Attachments: HDFS-6874-1.patch, HDFS-6874-branch-2.6.0.patch,
> HDFS-6874.02.patch, HDFS-6874.03.patch, HDFS-6874.patch
>
>
> GET_BLOCK_LOCATIONS operation is missing in HttpFS, which is already
> supported in WebHDFS. For the request of GETFILEBLOCKLOCATIONS in
> org.apache.hadoop.fs.http.server.HttpFSServer, BAD_REQUEST is returned so far:
> ...
> case GETFILEBLOCKLOCATIONS: {
> response = Response.status(Response.Status.BAD_REQUEST).build();
> break;
> }
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
