[jira] [Updated] (HDFS-11170) create API in filesystem public class to support assign parameter through builder
[ https://issues.apache.org/jira/browse/HDFS-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-11170: - Summary: create API in filesystem public class to support assign parameter through builder (was: Introduce builder style create API to FileSystem) > create API in filesystem public class to support assign parameter through > builder > -- > > Key: HDFS-11170 > URL: https://issues.apache.org/jira/browse/HDFS-11170 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > > FileSystem class supports multiple create functions to help user create file. > Some create functions has many parameters, it's hard for user to exactly > remember these parameters and their orders. This task is to add builder > based create functions to help user more easily create file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11170) Add create API in filesystem public class to support assign parameter through builder
[ https://issues.apache.org/jira/browse/HDFS-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-11170: - Summary: Add create API in filesystem public class to support assign parameter through builder (was: create API in filesystem public class to support assign parameter through builder) > Add create API in filesystem public class to support assign parameter through > builder > - > > Key: HDFS-11170 > URL: https://issues.apache.org/jira/browse/HDFS-11170 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > > FileSystem class supports multiple create functions to help user create file. > Some create functions has many parameters, it's hard for user to exactly > remember these parameters and their orders. This task is to add builder > based create functions to help user more easily create file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
[
https://issues.apache.org/jira/browse/HDFS-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pan Yuxuan updated HDFS-11167:
--
Attachment: HDFS-11167-0002.patch
fix shell check error.
> IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS
> "-jvm server"
> -
>
> Key: HDFS-11167
> URL: https://issues.apache.org/jira/browse/HDFS-11167
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
> Environment: IBM PowerPC
>Reporter: Pan Yuxuan
>Assignee: Pan Yuxuan
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11167-0001.patch, HDFS-11167-0002.patch
>
>
> When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
> error
> {noformat}
> jsvc error: Invalid JVM name specified server
> {noformat}
> This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
> For IBM JDK it is enough to run it without -jvm server.
> So I think we can check if IBM jdk in hdfs-config.sh before setting
> HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11170) Introduce builder style create API to FileSystem
[ https://issues.apache.org/jira/browse/HDFS-11170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HDFS-11170: - Summary: Introduce builder style create API to FileSystem (was: Refactor create API in filesystem public class to support assign parameter through builder) > Introduce builder style create API to FileSystem > > > Key: HDFS-11170 > URL: https://issues.apache.org/jira/browse/HDFS-11170 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: SammiChen >Assignee: SammiChen > > FileSystem class supports multiple create functions to help user create file. > Some create functions has many parameters, it's hard for user to exactly > remember these parameters and their orders. This task is to add builder > based create functions to help user more easily create file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10996) Ability to specify per-file EC policy at create time
[ https://issues.apache.org/jira/browse/HDFS-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689251#comment-15689251 ] SammiChen commented on HDFS-10996: -- Thanks Andrew for the clarification. I created JIRA HDFS-11170 for refactor create API. > Ability to specify per-file EC policy at create time > > > Key: HDFS-10996 > URL: https://issues.apache.org/jira/browse/HDFS-10996 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: SammiChen > Attachments: HDFS-10996-v1.patch, HDFS-10996-v2.patch, > HDFS-10996-v3.patch > > > Based on discussion in HDFS-10971, it would be useful to specify the EC > policy when the file is created. This is useful for situations where app > requirements do not map nicely to the current directory-level policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11170) Refactor create API in filesystem public class to support assign parameter through builder
SammiChen created HDFS-11170: Summary: Refactor create API in filesystem public class to support assign parameter through builder Key: HDFS-11170 URL: https://issues.apache.org/jira/browse/HDFS-11170 Project: Hadoop HDFS Issue Type: Improvement Reporter: SammiChen Assignee: SammiChen FileSystem class supports multiple create functions to help user create file. Some create functions has many parameters, it's hard for user to exactly remember these parameters and their orders. This task is to add builder based create functions to help user more easily create file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
[
https://issues.apache.org/jira/browse/HDFS-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689217#comment-15689217
]
Hadoop QA commented on HDFS-11167:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 3 new + 116 unchanged - 1 fixed =
119 total (was 117) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11167 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839972/HDFS-11167-0001.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux ad82554fdd0d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4667564 |
| shellcheck | v0.4.5 |
| shellcheck |
https://builds.apache.org/job/PreCommit-HDFS-Build/17645/artifact/patchprocess/diff-patch-shellcheck.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17645/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17645/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS
> "-jvm server"
> -
>
> Key: HDFS-11167
> URL: https://issues.apache.org/jira/browse/HDFS-11167
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
> Environment: IBM PowerPC
>Reporter: Pan Yuxuan
>Assignee: Pan Yuxuan
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11167-0001.patch
>
>
> When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
> error
> {noformat}
> jsvc error: Invalid JVM name specified server
> {noformat}
> This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
> For IBM JDK it is enough to run it without -jvm server.
> So I think we can check if IBM jdk in hdfs-config.sh before setting
> HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
[
https://issues.apache.org/jira/browse/HDFS-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pan Yuxuan updated HDFS-11167:
--
Fix Version/s: 3.0.0-alpha2
Status: Patch Available (was: Open)
> IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS
> "-jvm server"
> -
>
> Key: HDFS-11167
> URL: https://issues.apache.org/jira/browse/HDFS-11167
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
> Environment: IBM PowerPC
>Reporter: Pan Yuxuan
>Assignee: Pan Yuxuan
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-11167-0001.patch
>
>
> When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
> error
> {noformat}
> jsvc error: Invalid JVM name specified server
> {noformat}
> This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
> For IBM JDK it is enough to run it without -jvm server.
> So I think we can check if IBM jdk in hdfs-config.sh before setting
> HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11058) Implement 'hadoop fs -df' command for ViewFileSystem
[
https://issues.apache.org/jira/browse/HDFS-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15689032#comment-15689032
]
Manoj Govindassamy commented on HDFS-11058:
---
Test failure is not related to the patch.
> Implement 'hadoop fs -df' command for ViewFileSystem
> ---
>
> Key: HDFS-11058
> URL: https://issues.apache.org/jira/browse/HDFS-11058
> Project: Hadoop HDFS
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Labels: viewfs
> Attachments: HDFS-11058.01.patch, HDFS-11058.02.patch,
> HDFS-11058.03.patch, HDFS-11058.04.patch
>
>
> Df command doesn't seem to work well with ViewFileSystem. It always reports
> used data as 0. Here is the client mount table configuration I am using
> against a federated clusters of 2 NameNodes and 2 DataNoes.
> {code}
> 1
> 2
> 3
> 4 fs.defaultFS
> 5 viewfs://ClusterX/
> 6
> ..
> 11
> 12 fs.default.name
> 13 viewfs://ClusterX/
> 14
> ..
> 23
> 24 fs.viewfs.mounttable.ClusterX.link./nn0
> 25 hdfs://127.0.0.1:50001/
> 26
> 27
> 28 fs.viewfs.mounttable.ClusterX.link./nn1
> 29 hdfs://127.0.0.1:51001/
> 30
> 31
> 32 fs.viewfs.mounttable.ClusterX.link./nn2
> 33 hdfs://127.0.0.1:52001/nn2
> 34
> 35
> 36 fs.viewfs.mounttable.ClusterX.link./nn3
> 37 hdfs://127.0.0.1:52001/nn3
> 38
> 39
> 40 fs.viewfs.mounttable.ClusterY.linkMergeSlash
> 41 hdfs://127.0.0.1:50001/
> 42
> 43
> {code}
> {{Df}} command always reports Size/Available as 8.0E and the usage as 0 for
> any federated cluster.
> {noformat}
> # hadoop fs -fs viewfs://ClusterX/ -df /
> Filesystem Size UsedAvailable Use%
> viewfs://ClusterX/ 9223372036854775807 0 92233720368547758070%
> # hadoop fs -fs viewfs://ClusterX/ -df -h /
> Filesystem Size Used Available Use%
> viewfs://ClusterX/ 8.0 E 0 8.0 E0%
> # hadoop fs -fs viewfs://ClusterY/ -df -h /
> Filesystem Size Used Available Use%
> viewfs://ClusterY/ 8.0 E 0 8.0 E0%
> {noformat}
> Whereas {{Du}} command seems to work as expected even with ViewFileSystem.
> {noformat}
> # hadoop fs -fs viewfs://ClusterY/ -du -h /
> 10.6 K 31.8 K /build.log.16y
> 0 0 /user
> # hadoop fs -fs viewfs://ClusterX/ -du -h /
> 10.6 K 31.8 K /nn0
> 0 0 /nn1
> 20.2 K 35.8 K /nn3
> 40.6 K 34.3 K /nn4
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10368) Erasure Coding: Deprecate replication-related config keys
[
https://issues.apache.org/jira/browse/HDFS-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688897#comment-15688897
]
Hadoop QA commented on HDFS-10368:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 42 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 1s{color} | {color:orange} root: The patch generated 2 new + 1714 unchanged
- 16 fixed = 1716 total (was 1730) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
24s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
3s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10368 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840179/HDFS-10368-04.patch |
| Optional Tests | asflicense mvnsite compile javac javadoc mvninstall
unit findbugs checkstyle xml |
| uname | Linux 2d7d67bb79f0 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2bf9a15 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17643/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17643/testReport/ |
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688779#comment-15688779
]
Weiwei Yang commented on HDFS-11156:
Thanks a lot [~liuml07]
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11058) Implement 'hadoop fs -df' command for ViewFileSystem
[
https://issues.apache.org/jira/browse/HDFS-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688757#comment-15688757
]
Hadoop QA commented on HDFS-11058:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 9m 42s{color}
| {color:red} root generated 3 new + 711 unchanged - 3 fixed = 714 total (was
714) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
33s{color} | {color:green} root: The patch generated 0 new + 143 unchanged - 4
fixed = 143 total (was 147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
56s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZones |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11058 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840168/HDFS-11058.04.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 05dd024b5601 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2bf9a15 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/17642/artifact/patchprocess/diff-compile-javac-root.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17642/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17642/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688745#comment-15688745
]
Mingliang Liu commented on HDFS-11156:
--
It may happen this week or after the holiday. Thanks,
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10583) Add links to component's configuration UI page under Utilities dropdown
[
https://issues.apache.org/jira/browse/HDFS-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688706#comment-15688706
]
Hadoop QA commented on HDFS-10583:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 0m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10583 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821529/HDFS-10583.003.patch |
| Optional Tests | asflicense |
| uname | Linux dda9fb202f83 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2bf9a15 |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17644/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add links to component's configuration UI page under Utilities dropdown
> ---
>
> Key: HDFS-10583
> URL: https://issues.apache.org/jira/browse/HDFS-10583
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs, ui
>Affects Versions: 2.7.2
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Minor
> Attachments: HDFS-10583.001.patch, HDFS-10583.002.patch,
> HDFS-10583.003.patch, conf_link_on_DN.jpg, conf_link_on_NN.jpg
>
>
> When admin wants to explore some configuration properties, such as namenode
> and datanode, it will be helpful to provide an UI page to read them. This is
> extremely useful when nodes are having different configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10583) Add links to component's configuration UI page under Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-10583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688688#comment-15688688 ] Weiwei Yang commented on HDFS-10583: Hello [~vinayrpet] [~shahrs87] Sorry to send this small reminder, this patch was submitted several months ago and pretty straightforward fix. Would you please let me know if it makes sense to have this in 3.0 alpha 2? Appreciated for any of your comments. > Add links to component's configuration UI page under Utilities dropdown > --- > > Key: HDFS-10583 > URL: https://issues.apache.org/jira/browse/HDFS-10583 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, ui >Affects Versions: 2.7.2 >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Minor > Attachments: HDFS-10583.001.patch, HDFS-10583.002.patch, > HDFS-10583.003.patch, conf_link_on_DN.jpg, conf_link_on_NN.jpg > > > When admin wants to explore some configuration properties, such as namenode > and datanode, it will be helpful to provide an UI page to read them. This is > extremely useful when nodes are having different configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688666#comment-15688666
]
Weiwei Yang edited comment on HDFS-11156 at 11/23/16 2:37 AM:
--
Hello [~liuml07]
Would you help to review this patch ?
I have built latest trunk and tested with this patch, use
{code}
http://:/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS=0=1=yangww
{code}
it returns output
{code}
{"BlockLocations":[{"cachedHosts":[],"corrupt":false,"hosts":["deale1.fyre.ibm.com"],"length":134217728,"names":["9.30.167.122:9866"],"offset":0,"storageIds":["DS-2c136bcd-d658-4853-8b7d-9ecb354e6093"],"storageTypes":["DISK"],"topologyPaths":["/default-rack/9.30.167.122:9866"]}]}
{code}
client would easily parse this to BlockLocation array using json parser or
object mapper, just like the test case code I added to {{TestWebHDFS}}.
was (Author: cheersyang):
Hello Mingliang Liu
Would you help to review this patch ?
I have built latest trunk and tested with this patch, use
{code}
http://:/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS=0=1=yangww
{code}
it returns output
{code}
{"BlockLocations":[{"cachedHosts":[],"corrupt":false,"hosts":["deale1.fyre.ibm.com"],"length":134217728,"names":["9.30.167.122:9866"],"offset":0,"storageIds":["DS-2c136bcd-d658-4853-8b7d-9ecb354e6093"],"storageTypes":["DISK"],"topologyPaths":["/default-rack/9.30.167.122:9866"]}]}
{code}
client would easily parse this to BlockLocation array using json parser or
object mapper, just like the test case code I added to {{TestWebHDFS}}.
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688666#comment-15688666
]
Weiwei Yang commented on HDFS-11156:
Hello Mingliang Liu
Would you help to review this patch ?
I have built latest trunk and tested with this patch, use
{code}
http://:/webhdfs/v1/tmp/test1.img?op=GET_BLOCK_LOCATIONS=0=1=yangww
{code}
it returns output
{code}
{"BlockLocations":[{"cachedHosts":[],"corrupt":false,"hosts":["deale1.fyre.ibm.com"],"length":134217728,"names":["9.30.167.122:9866"],"offset":0,"storageIds":["DS-2c136bcd-d658-4853-8b7d-9ecb354e6093"],"storageTypes":["DISK"],"topologyPaths":["/default-rack/9.30.167.122:9866"]}]}
{code}
client would easily parse this to BlockLocation array using json parser or
object mapper, just like the test case code I added to {{TestWebHDFS}}.
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11168) Bump Netty 4 version
[ https://issues.apache.org/jira/browse/HDFS-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688622#comment-15688622 ] Andrew Wang commented on HDFS-11168: Hi Nicholas, * What is the target version for this JIRA? * Are the code changes due to API changes in Netty? If so, this might not be eligible for 2.x due to impact on downstream classpaths. * Also please do not use wildcard imports. > Bump Netty 4 version > > > Key: HDFS-11168 > URL: https://issues.apache.org/jira/browse/HDFS-11168 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h11168_20161122.patch > > > The current Netty 4 version is 4.1.0.Beta5. We should bump it to a non-beta > version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11168) Bump Netty 4 version
[
https://issues.apache.org/jira/browse/HDFS-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688611#comment-15688611
]
Hadoop QA commented on HDFS-11168:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 4s{color}
| {color:red} root generated 2 new + 714 unchanged - 0 fixed = 716 total (was
714) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 33s{color} | {color:orange} root: The patch generated 3 new + 12 unchanged -
3 fixed = 15 total (was 15) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
16s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}136m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.datanode.web.TestDatanodeHttpXFrame |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.web.webhdfs.TestParameterParser |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11168 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840159/h11168_20161122.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
[jira] [Commented] (HDFS-10368) Erasure Coding: Deprecate replication-related config keys
[ https://issues.apache.org/jira/browse/HDFS-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688592#comment-15688592 ] Rakesh R commented on HDFS-10368: - Thanks [~andrew.wang], I've rebased the previous patch on top of the latest trunk code and attached the same to jira. > Erasure Coding: Deprecate replication-related config keys > - > > Key: HDFS-10368 > URL: https://issues.apache.org/jira/browse/HDFS-10368 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: Rakesh R > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-10368-00.patch, HDFS-10368-01.patch, > HDFS-10368-02.patch, HDFS-10368-03.patch, HDFS-10368-04.patch > > > This jira is to visit the replication based config keys and deprecate them(if > necessary) in order to make it more meaningful. > Please refer [discussion > thread|https://issues.apache.org/jira/browse/HDFS-9869?focusedCommentId=15249363=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15249363] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10368) Erasure Coding: Deprecate replication-related config keys
[ https://issues.apache.org/jira/browse/HDFS-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-10368: Attachment: HDFS-10368-04.patch > Erasure Coding: Deprecate replication-related config keys > - > > Key: HDFS-10368 > URL: https://issues.apache.org/jira/browse/HDFS-10368 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: Rakesh R > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-10368-00.patch, HDFS-10368-01.patch, > HDFS-10368-02.patch, HDFS-10368-03.patch, HDFS-10368-04.patch > > > This jira is to visit the replication based config keys and deprecate them(if > necessary) in order to make it more meaningful. > Please refer [discussion > thread|https://issues.apache.org/jira/browse/HDFS-9869?focusedCommentId=15249363=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15249363] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-11154: Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Target Version/s: HDFS-7240 Status: Resolved (was: Patch Available) [~vagarychen] Thanks for contributing the patch. I have committed this to the ozone branch. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Fix For: HDFS-7240 > > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch, > HDFS-11154-HDFS-7240.007.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688566#comment-15688566
]
Weiwei Yang commented on HDFS-11156:
I don't think the test failures in latest jenkins job related to this patch, I
run them successfully locally, also 1st jenkins job UT was clean, v2/v3 patch
were just checkstyle fixes.
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11169) GetBlockLocations returns a block when offset > filesize and file only has 1 block
David Tucker created HDFS-11169: --- Summary: GetBlockLocations returns a block when offset > filesize and file only has 1 block Key: HDFS-11169 URL: https://issues.apache.org/jira/browse/HDFS-11169 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Environment: HDP 2.5, Ambari 2.4 Reporter: David Tucker Priority: Minor Start with a fresh deployment of HDFS. 1. Create a file. 2. AddBlock the file. 3. Call GetBlockLocations. Expectation: 0 blocks are returned because the only added block is incomplete. Observation: 1 block is returned. This only seems to occur when 1 block is in play (i.e. if you write a block and call AddBlock again, GetBlockLocations seems to behave as expected). This seems to be related to HDFS-513. I suspect the following line needs revision: https://github.com/apache/hadoop/blob/4484b48498b2ab2a40a404c487c7a4e875df10dc/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1062 I believe it should be >= instead of >: if (nrBlocks >= 0 && curBlk == nrBlocks) // offset >= end of file -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688543#comment-15688543 ] Anu Engineer commented on HDFS-11154: - +1, I will commit this shortly. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch, > HDFS-11154-HDFS-7240.007.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11154) Block Storage : store server state to persistent storage
[
https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688505#comment-15688505
]
Hadoop QA commented on HDFS-11154:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
51s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
55s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDelegationTokenFetcher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11154 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840158/HDFS-11154-HDFS-7240.007.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d63a8d9d42ae 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 334113a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17641/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17641/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17641/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/17641/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17641/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Block Storage : store server state to persistent
[jira] [Updated] (HDFS-11058) Implement 'hadoop fs -df' command for ViewFileSystem
[
https://issues.apache.org/jira/browse/HDFS-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HDFS-11058:
--
Attachment: HDFS-11058.04.patch
Thanks for the review [~andrew.wang]. Attaching v04 patch after the rebase to
latest trunk.
> Implement 'hadoop fs -df' command for ViewFileSystem
> ---
>
> Key: HDFS-11058
> URL: https://issues.apache.org/jira/browse/HDFS-11058
> Project: Hadoop HDFS
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Labels: viewfs
> Attachments: HDFS-11058.01.patch, HDFS-11058.02.patch,
> HDFS-11058.03.patch, HDFS-11058.04.patch
>
>
> Df command doesn't seem to work well with ViewFileSystem. It always reports
> used data as 0. Here is the client mount table configuration I am using
> against a federated clusters of 2 NameNodes and 2 DataNoes.
> {code}
> 1
> 2
> 3
> 4 fs.defaultFS
> 5 viewfs://ClusterX/
> 6
> ..
> 11
> 12 fs.default.name
> 13 viewfs://ClusterX/
> 14
> ..
> 23
> 24 fs.viewfs.mounttable.ClusterX.link./nn0
> 25 hdfs://127.0.0.1:50001/
> 26
> 27
> 28 fs.viewfs.mounttable.ClusterX.link./nn1
> 29 hdfs://127.0.0.1:51001/
> 30
> 31
> 32 fs.viewfs.mounttable.ClusterX.link./nn2
> 33 hdfs://127.0.0.1:52001/nn2
> 34
> 35
> 36 fs.viewfs.mounttable.ClusterX.link./nn3
> 37 hdfs://127.0.0.1:52001/nn3
> 38
> 39
> 40 fs.viewfs.mounttable.ClusterY.linkMergeSlash
> 41 hdfs://127.0.0.1:50001/
> 42
> 43
> {code}
> {{Df}} command always reports Size/Available as 8.0E and the usage as 0 for
> any federated cluster.
> {noformat}
> # hadoop fs -fs viewfs://ClusterX/ -df /
> Filesystem Size UsedAvailable Use%
> viewfs://ClusterX/ 9223372036854775807 0 92233720368547758070%
> # hadoop fs -fs viewfs://ClusterX/ -df -h /
> Filesystem Size Used Available Use%
> viewfs://ClusterX/ 8.0 E 0 8.0 E0%
> # hadoop fs -fs viewfs://ClusterY/ -df -h /
> Filesystem Size Used Available Use%
> viewfs://ClusterY/ 8.0 E 0 8.0 E0%
> {noformat}
> Whereas {{Du}} command seems to work as expected even with ViewFileSystem.
> {noformat}
> # hadoop fs -fs viewfs://ClusterY/ -du -h /
> 10.6 K 31.8 K /build.log.16y
> 0 0 /user
> # hadoop fs -fs viewfs://ClusterX/ -du -h /
> 10.6 K 31.8 K /nn0
> 0 0 /nn1
> 20.2 K 35.8 K /nn3
> 40.6 K 34.3 K /nn4
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11159) Add reencryptEDEK interface for KMS
[ https://issues.apache.org/jira/browse/HDFS-11159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11159: --- Issue Type: Improvement (was: Sub-task) Parent: (was: HDFS-10899) > Add reencryptEDEK interface for KMS > --- > > Key: HDFS-11159 > URL: https://issues.apache.org/jira/browse/HDFS-11159 > Project: Hadoop HDFS > Issue Type: Improvement > Components: kms >Reporter: Xiao Chen >Assignee: Xiao Chen > Attachments: HDFS-11159.01.patch > > > This is the KMS part. Please refer to HDFS-10899 for the design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11058) Implement 'hadoop fs -df' command for ViewFileSystem
[
https://issues.apache.org/jira/browse/HDFS-11058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688336#comment-15688336
]
Andrew Wang commented on HDFS-11058:
LGTM, +1 though it looks like this patch has gone stale. Could you rebase? I
promise to commit soon.
> Implement 'hadoop fs -df' command for ViewFileSystem
> ---
>
> Key: HDFS-11058
> URL: https://issues.apache.org/jira/browse/HDFS-11058
> Project: Hadoop HDFS
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Labels: viewfs
> Attachments: HDFS-11058.01.patch, HDFS-11058.02.patch,
> HDFS-11058.03.patch
>
>
> Df command doesn't seem to work well with ViewFileSystem. It always reports
> used data as 0. Here is the client mount table configuration I am using
> against a federated clusters of 2 NameNodes and 2 DataNoes.
> {code}
> 1
> 2
> 3
> 4 fs.defaultFS
> 5 viewfs://ClusterX/
> 6
> ..
> 11
> 12 fs.default.name
> 13 viewfs://ClusterX/
> 14
> ..
> 23
> 24 fs.viewfs.mounttable.ClusterX.link./nn0
> 25 hdfs://127.0.0.1:50001/
> 26
> 27
> 28 fs.viewfs.mounttable.ClusterX.link./nn1
> 29 hdfs://127.0.0.1:51001/
> 30
> 31
> 32 fs.viewfs.mounttable.ClusterX.link./nn2
> 33 hdfs://127.0.0.1:52001/nn2
> 34
> 35
> 36 fs.viewfs.mounttable.ClusterX.link./nn3
> 37 hdfs://127.0.0.1:52001/nn3
> 38
> 39
> 40 fs.viewfs.mounttable.ClusterY.linkMergeSlash
> 41 hdfs://127.0.0.1:50001/
> 42
> 43
> {code}
> {{Df}} command always reports Size/Available as 8.0E and the usage as 0 for
> any federated cluster.
> {noformat}
> # hadoop fs -fs viewfs://ClusterX/ -df /
> Filesystem Size UsedAvailable Use%
> viewfs://ClusterX/ 9223372036854775807 0 92233720368547758070%
> # hadoop fs -fs viewfs://ClusterX/ -df -h /
> Filesystem Size Used Available Use%
> viewfs://ClusterX/ 8.0 E 0 8.0 E0%
> # hadoop fs -fs viewfs://ClusterY/ -df -h /
> Filesystem Size Used Available Use%
> viewfs://ClusterY/ 8.0 E 0 8.0 E0%
> {noformat}
> Whereas {{Du}} command seems to work as expected even with ViewFileSystem.
> {noformat}
> # hadoop fs -fs viewfs://ClusterY/ -du -h /
> 10.6 K 31.8 K /build.log.16y
> 0 0 /user
> # hadoop fs -fs viewfs://ClusterX/ -du -h /
> 10.6 K 31.8 K /nn0
> 0 0 /nn1
> 20.2 K 35.8 K /nn3
> 40.6 K 34.3 K /nn4
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11168) Bump Netty 4 version
[ https://issues.apache.org/jira/browse/HDFS-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11168: --- Status: Patch Available (was: Open) > Bump Netty 4 version > > > Key: HDFS-11168 > URL: https://issues.apache.org/jira/browse/HDFS-11168 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h11168_20161122.patch > > > The current Netty 4 version is 4.1.0.Beta5. We should bump it to a non-beta > version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11168) Bump Netty 4 version
[ https://issues.apache.org/jira/browse/HDFS-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11168: --- Attachment: h11168_20161122.patch h11168_20161122.patch: bump Netty 4 to 4.1.6.Final. > Bump Netty 4 version > > > Key: HDFS-11168 > URL: https://issues.apache.org/jira/browse/HDFS-11168 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h11168_20161122.patch > > > The current Netty 4 version is 4.1.0.Beta5. We should bump it to a non-beta > version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.007.patch Fix the checkstyle warning > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch, > HDFS-11154-HDFS-7240.007.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11168) Bump Netty 4 version
Tsz Wo Nicholas Sze created HDFS-11168: -- Summary: Bump Netty 4 version Key: HDFS-11168 URL: https://issues.apache.org/jira/browse/HDFS-11168 Project: Hadoop HDFS Issue Type: Bug Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze The current Netty 4 version is 4.1.0.Beta5. We should bump it to a non-beta version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[
https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688247#comment-15688247
]
Hadoop QA commented on HDFS-11094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 10 new + 293 unchanged - 12 fixed = 303 total (was 305) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11094 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840122/HDFS-11094.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux ad91d461b11b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / beb70fe |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17639/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17639/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17639/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17639/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-10930) Refactor: Wrap Datanode IO related operations
[
https://issues.apache.org/jira/browse/HDFS-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688196#comment-15688196
]
Arpit Agarwal commented on HDFS-10930:
--
The findbugs issue looks valid. It can be addressed by restoring the close
calls in the {{finally}} block if the {{ris}} is null.
{code}
} finally {
if (ris != null) {
ris.close();
+ } else {
+IOUtils.closeStream(checksumIn);
+IOUtils.closeStream(blockIn);
}
{code}
The function will need some more changes to restore scoping for checksumIn and
blockIn.
> Refactor: Wrap Datanode IO related operations
> -
>
> Key: HDFS-10930
> URL: https://issues.apache.org/jira/browse/HDFS-10930
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-10930.01.patch, HDFS-10930.02.patch,
> HDFS-10930.03.patch, HDFS-10930.04.patch, HDFS-10930.05.patch,
> HDFS-10930.06.patch
>
>
> Datanode IO (Disk/Network) related operations and instrumentations are
> currently spilled in many classes such as DataNode.java, BlockReceiver.java,
> BlockSender.java, FsDatasetImpl.java, FsVolumeImpl.java,
> DirectoryScanner.java, BlockScanner.java, FsDatasetAsyncDiskService.java,
> LocalReplica.java, LocalReplicaPipeline.java, Storage.java, etc.
> This ticket is opened to consolidate IO related operations for easy
> instrumentation, metrics collection, logging and trouble shooting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11154) Block Storage : store server state to persistent storage
[
https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688172#comment-15688172
]
Hadoop QA commented on HDFS-11154:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
2s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.cblock.TestCBlockServerPersistence |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
| | hadoop.hdfs.server.datanode.TestBlockRecovery |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11154 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840105/HDFS-11154-HDFS-7240.006.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 80ee7f6a0b3b 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 334113a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
[jira] [Commented] (HDFS-11145) Implement getTrashRoot() for ViewFileSystem
[ https://issues.apache.org/jira/browse/HDFS-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688110#comment-15688110 ] Hudson commented on HDFS-11145: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10876 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10876/]) HDFS-11145. Implement getTrashRoot() for ViewFileSystem. (Manoj (lei: rev 3a09e5970db7e6db4cc47f18dafc1daf0ff2538b) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/viewfs/ViewFileSystemBaseTest.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/viewfs/TestViewFileSystemHdfs.java > Implement getTrashRoot() for ViewFileSystem > --- > > Key: HDFS-11145 > URL: https://issues.apache.org/jira/browse/HDFS-11145 > Project: Hadoop HDFS > Issue Type: Task >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Labels: viewfs > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-11145.01.patch > > > ViewFileSystem doesn't have the custom implementation of > FileSystem#getTrashRoot(Path) yet and hence irrespective of Paths passed in, > ViewFileSystem always returns the user specific .Trash directory. > ViewFileSystem should implement getTrashRoot(Path) and delegate the call to > the respective mounted file system which can then examine about EZ or other > criteria and return a proper Trash directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11143) start-dfs.sh doesn't return any error message even namenode is not up.
[ https://issues.apache.org/jira/browse/HDFS-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated HDFS-11143: Summary: start-dfs.sh doesn't return any error message even namenode is not up. (was: start.sh doesn't return any error message even namenode is not up.) > start-dfs.sh doesn't return any error message even namenode is not up. > -- > > Key: HDFS-11143 > URL: https://issues.apache.org/jira/browse/HDFS-11143 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11143) start.sh doesn't return any error message even namenode is not up.
[ https://issues.apache.org/jira/browse/HDFS-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688053#comment-15688053 ] Yufei Gu commented on HDFS-11143: - It's "start-dfs.sh" in trunk. > start.sh doesn't return any error message even namenode is not up. > -- > > Key: HDFS-11143 > URL: https://issues.apache.org/jira/browse/HDFS-11143 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11143) start.sh doesn't return any error message even namenode is not up.
[ https://issues.apache.org/jira/browse/HDFS-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15688021#comment-15688021 ] Allen Wittenauer commented on HDFS-11143: - What is "start.sh"? Is this trunk or branch-2? > start.sh doesn't return any error message even namenode is not up. > -- > > Key: HDFS-11143 > URL: https://issues.apache.org/jira/browse/HDFS-11143 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11145) Implement getTrashRoot() for ViewFileSystem
[ https://issues.apache.org/jira/browse/HDFS-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-11145: - Resolution: Fixed Fix Version/s: 3.0.0-alpha2 Status: Resolved (was: Patch Available) +1. LGTM. Committed to trunk. Thanks [~manojg]. > Implement getTrashRoot() for ViewFileSystem > --- > > Key: HDFS-11145 > URL: https://issues.apache.org/jira/browse/HDFS-11145 > Project: Hadoop HDFS > Issue Type: Task >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Labels: viewfs > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-11145.01.patch > > > ViewFileSystem doesn't have the custom implementation of > FileSystem#getTrashRoot(Path) yet and hence irrespective of Paths passed in, > ViewFileSystem always returns the user specific .Trash directory. > ViewFileSystem should implement getTrashRoot(Path) and delegate the call to > the respective mounted file system which can then examine about EZ or other > criteria and return a proper Trash directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11162) Block Storage : add command line tool
[ https://issues.apache.org/jira/browse/HDFS-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11162: -- Attachment: HDFS-11162-HDFS-7240.001.patch > Block Storage : add command line tool > - > > Key: HDFS-11162 > URL: https://issues.apache.org/jira/browse/HDFS-11162 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11162-HDFS-7240.001.patch > > > The block storage service uses a client side command line tool for volume > create/delete/info/list. This JIRA it to add this command line tool. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10368) Erasure Coding: Deprecate replication-related config keys
[ https://issues.apache.org/jira/browse/HDFS-10368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687974#comment-15687974 ] Andrew Wang commented on HDFS-10368: Sorry for leaving this sitting for so long, looks like the patch no longer applies cleanly. I'm +1 pending a fresh rebase though, thanks again for working on this Rakesh! > Erasure Coding: Deprecate replication-related config keys > - > > Key: HDFS-10368 > URL: https://issues.apache.org/jira/browse/HDFS-10368 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: Rakesh R > Labels: hdfs-ec-3.0-must-do > Attachments: HDFS-10368-00.patch, HDFS-10368-01.patch, > HDFS-10368-02.patch, HDFS-10368-03.patch > > > This jira is to visit the replication based config keys and deprecate them(if > necessary) in order to make it more meaningful. > Please refer [discussion > thread|https://issues.apache.org/jira/browse/HDFS-9869?focusedCommentId=15249363=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15249363] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
[
https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687954#comment-15687954
]
Hadoop QA commented on HDFS-11160:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 105 unchanged - 1 fixed = 106 total (was 106) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| Timed out junit tests | org.apache.hadoop.hdfs.TestReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11160 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840065/HDFS-11160.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8c6814114519 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 83cc726 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17637/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17637/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17637/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17637/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> VolumeScanner reports write-in-progress replicas as corrupt incorrectly
>
[jira] [Updated] (HDFS-11094) Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter
[ https://issues.apache.org/jira/browse/HDFS-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated HDFS-11094: --- Attachment: HDFS-11094.008.patch [~liuml07], attaching a patch that includes unit tests both on the DN and NN side of the change. I mocked out most of it, so the tests should be pretty simple. But a review to make sure that I'm testing what I think I'm testing would be appreciated. > Send back HAState along with NamespaceInfo during a versionRequest as an > optional parameter > --- > > Key: HDFS-11094 > URL: https://issues.apache.org/jira/browse/HDFS-11094 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: HDFS-11094.001.patch, HDFS-11094.002.patch, > HDFS-11094.003.patch, HDFS-11094.004.patch, HDFS-11094.005.patch, > HDFS-11094.006.patch, HDFS-11094.007.patch, HDFS-11094.008.patch > > > The datanode should know which NN is active when it is connecting/registering > to the NN. Currently, it only figures this out during its first (and > subsequent) heartbeat(s) and so there is a period of time where the datanode > is alive and registered, but can't actually do anything because it doesn't > know which NN is active. A byproduct of this is that the MiniDFSCluster will > become active before it knows what NN is active, which can lead to NPEs when > calling getActiveNN(). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.006.patch The cblock test persistence test fail is related. Fix in v006 patch > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch, HDFS-11154-HDFS-7240.006.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687766#comment-15687766
]
Hadoop QA commented on HDFS-11164:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
| | hadoop.hdfs.TestBlocksScheduledCounter |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11164 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840063/HDFS-11164-01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux abe1c66e33cd 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / afcf8d3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
[
https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687759#comment-15687759
]
Hadoop QA commented on HDFS-11160:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 105 unchanged - 1 fixed = 106 total (was 106) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 50s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.fs.viewfs.TestViewFsHdfs |
| | hadoop.fs.TestSymlinkHdfsFileContext |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.TestReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11160 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840065/HDFS-11160.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c73454c14209 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / afcf8d3 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17634/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17634/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17634/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11154) Block Storage : store server state to persistent storage
[
https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687719#comment-15687719
]
Hadoop QA commented on HDFS-11154:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
51s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-7240 has 10
extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 3s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.cblock.TestCBlockServerPersistence |
| | hadoop.hdfs.tools.TestDelegationTokenFetcher |
| | hadoop.ozone.TestStorageContainerManager |
| | hadoop.ozone.container.common.TestDatanodeStateMachine |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:e809691 |
| JIRA Issue | HDFS-11154 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840072/HDFS-11154-HDFS-7240.005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 35630585aa7d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 334113a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/17636/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17636/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17636/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/17636/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
[jira] [Commented] (HDFS-11148) Update DataNode to use StorageLocationChecker at startup
[
https://issues.apache.org/jira/browse/HDFS-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687623#comment-15687623
]
Hudson commented on HDFS-11148:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10873 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10873/])
HDFS-11148. Update DataNode to use StorageLocationChecker at startup. (arp: rev
613b902b9808c1c679674047cff15feade01dfab)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailure.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/InternalDataNodeTestUtils.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/checker/TestStorageLocationChecker.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeUUID.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
> Update DataNode to use StorageLocationChecker at startup
>
>
> Key: HDFS-11148
> URL: https://issues.apache.org/jira/browse/HDFS-11148
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.0.0-alpha2
>
>
> The DataNode can use the {{StorageLocationChecker}} introduced by HDFS-9
> to parallelize checking Storage Locations at process startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11148) Update DataNode to use StorageLocationChecker at startup
[
https://issues.apache.org/jira/browse/HDFS-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-11148:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Pushed to trunk. Thank you for the code reviews [~anu].
> Update DataNode to use StorageLocationChecker at startup
>
>
> Key: HDFS-11148
> URL: https://issues.apache.org/jira/browse/HDFS-11148
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 3.0.0-alpha2
>
>
> The DataNode can use the {{StorageLocationChecker}} introduced by HDFS-9
> to parallelize checking Storage Locations at process startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11148) Update DataNode to use StorageLocationChecker at startup
[
https://issues.apache.org/jira/browse/HDFS-11148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687524#comment-15687524
]
ASF GitHub Bot commented on HDFS-11148:
---
Github user anuengineer commented on the issue:
https://github.com/apache/hadoop/pull/162
Arpit, thanks for updating the patch. +1. LGTM.
> Update DataNode to use StorageLocationChecker at startup
>
>
> Key: HDFS-11148
> URL: https://issues.apache.org/jira/browse/HDFS-11148
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>
> The DataNode can use the {{StorageLocationChecker}} introduced by HDFS-9
> to parallelize checking Storage Locations at process startup.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11154) Block Storage : store server state to persistent storage
[ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11154: -- Attachment: HDFS-11154-HDFS-7240.005.patch The v005 patch was somehow checked by Jenkins twice. The first run was fine, all the warnings were unrelated. The second run complained about a number of things but the report links are all dead. So I suspect it hit some Jenkins internal issues in the second run. Upload v005 patch a second time to trigger one more Jenkins check. > Block Storage : store server state to persistent storage > > > Key: HDFS-11154 > URL: https://issues.apache.org/jira/browse/HDFS-11154 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11154-HDFS-7240.001.patch, > HDFS-11154-HDFS-7240.002.patch, HDFS-11154-HDFS-7240.003.patch, > HDFS-11154-HDFS-7240.004.patch, HDFS-11154-HDFS-7240.005.patch, > HDFS-11154-HDFS-7240.005.patch > > > Currently, all the storage state are kept in server memory. If server > crashes, we would lose all the volume information. This JIRA stores server > internal state into its local disk. Such that on server failure, we can > simply restart server and restore volume information from disk. > More specifically, the internal state written to disk is mainly the mapping > from volume to its underlying containers, plus some meta information such as > volume size, block size, etc. > Note that this is only a simple, minimum set mechanism for persistence. It is > more like a counterpart of fsimage in HDFS, but without edit logs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10918) Add a tool to get FileEncryptionInfo from CLI
[ https://issues.apache.org/jira/browse/HDFS-10918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687444#comment-15687444 ] Yongjun Zhang commented on HDFS-10918: -- Hi [~maniraj...@gmail.com] and [~xiaochen], I did run into the same failure, I did do "mvn clean" but for some reason the old HdfsAdmin.class file was not cleaned up. I tried quite a few times and the problem remains. A later attempt did clean it up though. Thanks. > Add a tool to get FileEncryptionInfo from CLI > - > > Key: HDFS-10918 > URL: https://issues.apache.org/jira/browse/HDFS-10918 > Project: Hadoop HDFS > Issue Type: New Feature > Components: encryption >Reporter: Xiao Chen >Assignee: Xiao Chen > Labels: supportability > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10918.01.patch, HDFS-10918.02.patch, > HDFS-10918.03.patch, HDFS-10918.04.patch, HDFS-10918.05.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11164:
Description:
When mover is trying to move a pinned block to another datanode, it will
internally hits the following IOException and mark the block movement as
{{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
will continue moving this block until it reaches {{retryMaxAttempts}}. If the
block movement failure(s) are only due to block pinning, then retry is
unnecessary. The idea of this jira is to avoid retry attempts of pinned blocks
as they won't be able to move to a different node.
{code}
2016-11-22 10:56:10,537 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher:
Failed to move blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
exception java.io.IOException: Got error, status=ERROR, status message Not able
to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy block
BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
/127.0.0.1:19501, reportedBlock move is failed
at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
was:
When mover is trying to move a pinned block to another datanode, it will
internally hits the following IOException and mark the block movement as
{{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
will continue moving this block until it reaches {{retryMaxAttempts}}. This
retry is unnecessary and would be good to avoid retry attempts as pinned block
won't be able to move.
{code}
2016-11-22 10:56:10,537 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher:
Failed to move blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
exception java.io.IOException: Got error, status=ERROR, status message Not able
to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy block
BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
/127.0.0.1:19501, reportedBlock move is failed
at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
at
org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch, HDFS-11164-01.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. If the
> block movement failure(s) are only due to block pinning, then retry is
> unnecessary. The idea of this jira is to avoid retry attempts of pinned
> blocks as they won't be able to move to a different node.
> {code}
> 2016-11-22
[jira] [Commented] (HDFS-11163) Mover should move the file blocks to default storage once policy is unset
[
https://issues.apache.org/jira/browse/HDFS-11163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687430#comment-15687430
]
Arpit Agarwal commented on HDFS-11163:
--
Nice find [~surendrasingh].
> Mover should move the file blocks to default storage once policy is unset
> -
>
> Key: HDFS-11163
> URL: https://issues.apache.org/jira/browse/HDFS-11163
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Affects Versions: 2.8.0
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>
> HDFS-9534 added new API in FileSystem to unset the storage policy. Once
> policy is unset blocks should move back to the default storage policy.
> Currently mover is not moving file blocks which have zero storage ID
> {code}
> // currently we ignore files with unspecified storage policy
> if (policyId == HdfsConstants.BLOCK_STORAGE_POLICY_ID_UNSPECIFIED) {
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11112) Journal Nodes should refuse to format non-empty directories
[
https://issues.apache.org/jira/browse/HDFS-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687419#comment-15687419
]
Arpit Agarwal commented on HDFS-2:
--
Sorry I couldn't respond earlier. [~jojochuang] that is correct, if the format
request comes in via RPC then the JN skips the check for non-empty directories.
> Journal Nodes should refuse to format non-empty directories
> ---
>
> Key: HDFS-2
> URL: https://issues.apache.org/jira/browse/HDFS-2
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Yiqun Lin
> Attachments: HDFS-2.001.patch, HDFS-2.002.patch
>
>
> Journal Nodes should reject the {{format}} RPC request if a storage directory
> is non-empty. The relevant code is in {{JNStorage#format}}.
> {code}
> void format(NamespaceInfo nsInfo) throws IOException {
> setStorageInfo(nsInfo);
> ...
> unlockAll();
> sd.clearDirectory();
> writeProperties(sd);
> createPaxosDir();
> analyzeStorage();
> {code}
> This would make the behavior similar to {{namenode -format -nonInteractive}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11160) VolumeScanner reports write-in-progress replicas as corrupt incorrectly
[ https://issues.apache.org/jira/browse/HDFS-11160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-11160: --- Attachment: HDFS-11160.002.patch Submit 002 patch to fix test failures. > VolumeScanner reports write-in-progress replicas as corrupt incorrectly > --- > > Key: HDFS-11160 > URL: https://issues.apache.org/jira/browse/HDFS-11160 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode > Environment: CDH5.7.4 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang > Attachments: HDFS-11160.001.patch, HDFS-11160.002.patch, > HDFS-11160.reproduce.patch > > > Due to a race condition initially reported in HDFS-6804, VolumeScanner may > erroneously detect good replicas as corrupt. This is serious because in some > cases it results in data loss if all replicas are declared corrupt. > We are investigating an incidence that caused very high block corruption rate > in a relatively small cluster. Initially, we thought HDFS-11056 is to blame. > However, after applying HDFS-11056, we are still seeing VolumeScanner > reporting corrupt replicas. > It turns out that if a replica is being appended while VolumeScanner is > scanning it, VolumeScanner may use the new checksum to compare against old > data, causing checksum mismatch. > I have a unit test to reproduce the error. Will attach later. > To fix it, I propose a FinalizedReplica object should also have a > lastChecksum field like ReplicaBeingWritten, and BlockSender should use the > in-memory lastChecksum to verify the partial data in the last chunk on disk. > File this jira to discuss a good fix for this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11164:
Attachment: HDFS-11164-01.patch
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch, HDFS-11164-01.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. This
> retry is unnecessary and would be good to avoid retry attempts as pinned
> block won't be able to move.
> {code}
> 2016-11-22 10:56:10,537 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
> exception java.io.IOException: Got error, status=ERROR, status message Not
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
> /127.0.0.1:19501, reportedBlock move is failed
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11143) start.sh doesn't return any error message even namenode is not up.
[ https://issues.apache.org/jira/browse/HDFS-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687349#comment-15687349 ] Yufei Gu commented on HDFS-11143: - Yes, Hadoop management providers can start dfs gracefully, but the script start.sh is for the people who don't use Hadoop management software. > start.sh doesn't return any error message even namenode is not up. > -- > > Key: HDFS-11143 > URL: https://issues.apache.org/jira/browse/HDFS-11143 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11091) Implement a getTrashRoot that does not fall-back
[
https://issues.apache.org/jira/browse/HDFS-11091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687312#comment-15687312
]
Xiao Chen commented on HDFS-11091:
--
Hi Yuanbo,
What we meant in HDFS-10756 is NOT changing current behavior. This is because
even a RTE won't have incompatible signature, it's still an incompatible
behavior. Rather, adding an overload (or a new method with more explicit name)
that throws may be better.
Thanks.
> Implement a getTrashRoot that does not fall-back
>
>
> Key: HDFS-11091
> URL: https://issues.apache.org/jira/browse/HDFS-11091
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.8.0
>Reporter: Xiao Chen
>Assignee: Yuanbo Liu
>
> From HDFS-10756's
> [discussion|https://issues.apache.org/jira/browse/HDFS-10756?focusedCommentId=15623755=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15623755]:
> {{getTrashRoot}} is supposed to return the trash dir considering encryption
> zone. But if there's an error encountered (e.g. access control exception), it
> falls back to the default trash dir.
> Although there is a warning message about this, it is still a somewhat
> surprising behavior. The fall back was added by HDFS-9799 for compatibility
> reasons. This jira is to propose we add a getTrashRoot that throws, which
> will actually be more user-friendly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687118#comment-15687118
]
Hadoop QA commented on HDFS-11156:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 231
unchanged - 4 fixed = 231 total (was 235) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 65m 37s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.server.balancer.TestBalancer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11156 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840030/HDFS-11156.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 1fc06549b7a4 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a926f89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17633/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17633/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client
[jira] [Commented] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686994#comment-15686994
]
Hadoop QA commented on HDFS-11164:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
192 unchanged - 0 fixed = 194 total (was 192) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
3s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.web.TestWebHDFS |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11164 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840025/HDFS-11164-00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 277d562c116d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a926f89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17632/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
|
[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11156:
---
Attachment: HDFS-11156.03.patch
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch,
> HDFS-11156.03.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11164:
Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1)
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. This
> retry is unnecessary and would be good to avoid retry attempts as pinned
> block won't be able to move.
> {code}
> 2016-11-22 10:56:10,537 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
> exception java.io.IOException: Got error, status=ERROR, status message Not
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
> /127.0.0.1:19501, reportedBlock move is failed
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11164:
Status: Patch Available (was: Open)
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. This
> retry is unnecessary and would be good to avoid retry attempts as pinned
> block won't be able to move.
> {code}
> 2016-11-22 10:56:10,537 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
> exception java.io.IOException: Got error, status=ERROR, status message Not
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
> /127.0.0.1:19501, reportedBlock move is failed
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11164:
Target Version/s: 3.0.0-alpha1
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. This
> retry is unnecessary and would be good to avoid retry attempts as pinned
> block won't be able to move.
> {code}
> 2016-11-22 10:56:10,537 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
> exception java.io.IOException: Got error, status=ERROR, status message Not
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
> /127.0.0.1:19501, reportedBlock move is failed
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686773#comment-15686773
]
Rakesh R commented on HDFS-11164:
-
Attached an initial attempt by introducing {{ERROR_BLOCK_PINNED}} status code.
Here the Mover tool will check that the block movement failure(s) are only due
to block pinning. If yes, just mark as failed and exit without any retries.
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. This
> retry is unnecessary and would be good to avoid retry attempts as pinned
> block won't be able to move.
> {code}
> 2016-11-22 10:56:10,537 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
> exception java.io.IOException: Got error, status=ERROR, status message Not
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
> /127.0.0.1:19501, reportedBlock move is failed
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11164) Mover should avoid unnecessary retries if the block is pinned
[
https://issues.apache.org/jira/browse/HDFS-11164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-11164:
Attachment: HDFS-11164-00.patch
> Mover should avoid unnecessary retries if the block is pinned
> -
>
> Key: HDFS-11164
> URL: https://issues.apache.org/jira/browse/HDFS-11164
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-11164-00.patch
>
>
> When mover is trying to move a pinned block to another datanode, it will
> internally hits the following IOException and mark the block movement as
> {{failure}}. Since the Mover has {{dfs.mover.retry.max.attempts}} configs, it
> will continue moving this block until it reaches {{retryMaxAttempts}}. This
> retry is unnecessary and would be good to avoid retry attempts as pinned
> block won't be able to move.
> {code}
> 2016-11-22 10:56:10,537 WARN
> org.apache.hadoop.hdfs.server.balancer.Dispatcher: Failed to move
> blk_1073741825_1001 with size=52 from 127.0.0.1:19501:DISK to
> 127.0.0.1:19758:ARCHIVE through 127.0.0.1:19501
> java.io.IOException: Got error, status=ERROR, status message opReplaceBlock
> BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 received
> exception java.io.IOException: Got error, status=ERROR, status message Not
> able to copy block 1073741825 to /127.0.0.1:19826 because it's pinned , copy
> block BP-1772076264-10.252.146.200-1479792322960:blk_1073741825_1001 from
> /127.0.0.1:19501, reportedBlock move is failed
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:118)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:417)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:358)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$5(Dispatcher.java:322)
> at
> org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:1075)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686763#comment-15686763
]
Hadoop QA commented on HDFS-11156:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
230 unchanged - 4 fixed = 231 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11156 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840009/HDFS-11156.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 60cd3d32f1a4 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a926f89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17631/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/17631/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17631/testReport/ |
| modules | C:
[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11156:
---
Attachment: HDFS-11156.02.patch
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch, HDFS-11156.02.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10994) Support an XOR policy XOR-2-1-64k in HDFS
[
https://issues.apache.org/jira/browse/HDFS-10994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686343#comment-15686343
]
Hadoop QA commented on HDFS-10994:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 36s{color} | {color:orange} root: The patch generated 33 new + 126 unchanged
- 6 fixed = 159 total (was 132) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
24s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
6s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.hdfs.TestDFSClientRetries |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-10994 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839954/HDFS-10994-v4.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 46c604c5f479 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29
20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6f80742 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/17628/artifact/patchprocess/diff-checkstyle-root.txt
|
| unit |
[jira] [Commented] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686250#comment-15686250
]
Hadoop QA commented on HDFS-11156:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
231 unchanged - 3 fixed = 234 total (was 234) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 61m
54s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 9s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Private method
org.apache.hadoop.hdfs.web.JsonUtil.toJsonArray(DatanodeInfo[]) is never called
At JsonUtil.java:called At JsonUtil.java:[lines 189-198] |
| | Private method
org.apache.hadoop.hdfs.web.JsonUtil.toJsonMap(ExtendedBlock) is never called
At JsonUtil.java:called At JsonUtil.java:[lines 139-148] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11156 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839964/HDFS-11156.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 99268116dc01 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6f80742 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Updated] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
[
https://issues.apache.org/jira/browse/HDFS-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pan Yuxuan updated HDFS-11167:
--
Attachment: HDFS-11167-0001.patch
Attach a simple patch for this.
> IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS
> "-jvm server"
> -
>
> Key: HDFS-11167
> URL: https://issues.apache.org/jira/browse/HDFS-11167
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
> Environment: IBM PowerPC
>Reporter: Pan Yuxuan
>Assignee: Pan Yuxuan
> Attachments: HDFS-11167-0001.patch
>
>
> When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
> error
> {noformat}
> jsvc error: Invalid JVM name specified server
> {noformat}
> This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
> For IBM JDK it is enough to run it without -jvm server.
> So I think we can check if IBM jdk in hdfs-config.sh before setting
> HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
[
https://issues.apache.org/jira/browse/HDFS-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pan Yuxuan updated HDFS-11167:
--
Description:
When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
error
{noformat}
jsvc error: Invalid JVM name specified server
{noformat}
This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
For IBM JDK it is enough to run it without -jvm server.
So I think we can check if IBM jdk in hdfs-config.sh before setting
HDFS_DATANODE_SECURE_EXTRA_OPTS.
was:
When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
error
{{noformat}}
jsvc error: Invalid JVM name specified server
{{noformat}}
This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
For IBM JDK it is enough to run it without -jvm server.
So I think we can check if IBM jdk in hdfs-config.sh before setting
HDFS_DATANODE_SECURE_EXTRA_OPTS.
> IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS
> "-jvm server"
> -
>
> Key: HDFS-11167
> URL: https://issues.apache.org/jira/browse/HDFS-11167
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 3.0.0-alpha1
> Environment: IBM PowerPC
>Reporter: Pan Yuxuan
>Assignee: Pan Yuxuan
>
> When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
> error
> {noformat}
> jsvc error: Invalid JVM name specified server
> {noformat}
> This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
> For IBM JDK it is enough to run it without -jvm server.
> So I think we can check if IBM jdk in hdfs-config.sh before setting
> HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11166) Doc for GET_BLOCK_LOCATIONS is missing in webhdfs API
[ https://issues.apache.org/jira/browse/HDFS-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686154#comment-15686154 ] Weiwei Yang commented on HDFS-11166: HDFS-11156 will change the format of the GET_BLOCK_LOCATIONS output, will add doc after HDFS-11156 is done. > Doc for GET_BLOCK_LOCATIONS is missing in webhdfs API > - > > Key: HDFS-11166 > URL: https://issues.apache.org/jira/browse/HDFS-11166 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Affects Versions: 2.7.3 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > Add GET_BLOCK_LOCATIONS in webhdfs API docs, currently it is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
Pan Yuxuan created HDFS-11167:
-
Summary: IBM java can not start secure datanode with
HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
Key: HDFS-11167
URL: https://issues.apache.org/jira/browse/HDFS-11167
Project: Hadoop HDFS
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0-alpha1
Environment: IBM PowerPC
Reporter: Pan Yuxuan
Assignee: Pan Yuxuan
When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
error
{{noformat}}
jsvc error: Invalid JVM name specified server
{{noformat}}
This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
For IBM JDK it is enough to run it without -jvm server.
So I think we can check if IBM jdk in hdfs-config.sh before setting
HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11166) Doc for GET_BLOCK_LOCATIONS is missing in webhdfs API
Weiwei Yang created HDFS-11166: -- Summary: Doc for GET_BLOCK_LOCATIONS is missing in webhdfs API Key: HDFS-11166 URL: https://issues.apache.org/jira/browse/HDFS-11166 Project: Hadoop HDFS Issue Type: Bug Components: documentation, webhdfs Affects Versions: 2.7.3 Reporter: Weiwei Yang Assignee: Weiwei Yang Add GET_BLOCK_LOCATIONS in webhdfs API docs, currently it is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11143) start.sh doesn't return any error message even namenode is not up.
[ https://issues.apache.org/jira/browse/HDFS-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686112#comment-15686112 ] Weiwei Yang commented on HDFS-11143: I don't think Hadoop scripts really handle that much, it mostly was handled by higher level management providers, such as ambari. So this is not a bug. This is good to have, as a more general task than just namenode. > start.sh doesn't return any error message even namenode is not up. > -- > > Key: HDFS-11143 > URL: https://issues.apache.org/jira/browse/HDFS-11143 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Yufei Gu > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-11165) ErasureCodeBenchmarkThroughput report "out of heap memory" during execution
SammiChen created HDFS-11165: Summary: ErasureCodeBenchmarkThroughput report "out of heap memory" during execution Key: HDFS-11165 URL: https://issues.apache.org/jira/browse/HDFS-11165 Project: Hadoop HDFS Issue Type: Bug Reporter: SammiChen When run hadoop jar ./share/hadoop/hdfs/hadoop-hdfs-3.0.0-alpha2-SNAPSHOT-tests.jar org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput read 5000 ec 5 stf, following error happens, Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.doBenchmark(ErasureCodeBenchmarkThroughput.java:136) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.benchmark(ErasureCodeBenchmarkThroughput.java:161) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.run(ErasureCodeBenchmarkThroughput.java:261) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.main(ErasureCodeBenchmarkThroughput.java:422) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:239) at org.apache.hadoop.util.RunJar.main(RunJar.java:153) Caused by: java.lang.OutOfMemoryError: Java heap space -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11156:
---
Status: Patch Available (was: In Progress)
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11156) Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
[
https://issues.apache.org/jira/browse/HDFS-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HDFS-11156:
---
Attachment: HDFS-11156.01.patch
> Webhdfs rest api GET_BLOCK_LOCATIONS output doesn't comply with FileSystem API
> --
>
> Key: HDFS-11156
> URL: https://issues.apache.org/jira/browse/HDFS-11156
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.7.3
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HDFS-11156.01.patch
>
>
> Following webhdfs REST API
> {code}
> http://:/webhdfs/v1/?op=GET_BLOCK_LOCATIONS=0=1
> {code}
> will get a response like
> {code}
> {
> "LocatedBlocks" : {
> "fileLength" : 1073741824,
> "isLastBlockComplete" : true,
> "isUnderConstruction" : false,
> "lastLocatedBlock" : { ... },
> "locatedBlocks" : [ {...} ]
> }
> }
> {code}
> This represents for *o.a.h.h.p.LocatedBlocks*. However according to
> *FileSystem* API,
> {code}
> public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
> {code}
> clients would expect an array of BlockLocation. This mismatch should be
> fixed. Marked as Incompatible change as this will change the output of the
> GET_BLOCK_LOCATIONS API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11152) Start erasure coding policy ID number from 1 instead of 0 to void potential unexpected errors
[
https://issues.apache.org/jira/browse/HDFS-11152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15686035#comment-15686035
]
Hadoop QA commented on HDFS-11152:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HDFS-11152 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839956/HDFS-11152-v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5ad978252f29 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 6f80742 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/17629/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/17629/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Start erasure coding policy ID number from 1 instead of 0 to void potential
> unexpected errors
> -
>
> Key: HDFS-11152
> URL: https://issues.apache.org/jira/browse/HDFS-11152
> Project: Hadoop HDFS
> Issue Type: Task
>Affects Versions: 3.0.0-alpha1
>Reporter: SammiChen
>Assignee: SammiChen
> Labels:
