[jira] [Commented] (HDFS-12603) Enable async edit logging by default
[
https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194181#comment-16194181
]
Hadoop QA commented on HDFS-12603:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 49s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12603 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890602/HDFS-12603.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 1ba14396e9fe 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 25f31d9 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[
https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194176#comment-16194176
]
Hadoop QA commented on HDFS-12582:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
290 unchanged - 3 fixed = 292 total (was 293) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 19s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12582 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890641/HDFS-12582.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux abef255f02d5 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 25f31d9 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Updated] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12608: -- Attachment: HDFS-12608-HDFS-7240.00.patch > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12608-HDFS-7240.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12608: -- Attachment: (was: HDFS-12608.00.patch) > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12608-HDFS-7240.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12608) Remove Warnings when building ozone
[
https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194165#comment-16194165
]
Hadoop QA commented on HDFS-12608:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-12608 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-12608 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890651/HDFS-12608.00.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21559/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Remove Warnings when building ozone
> ---
>
> Key: HDFS-12608
> URL: https://issues.apache.org/jira/browse/HDFS-12608
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
> Attachments: HDFS-12608.00.patch
>
>
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @
> org.apache.hadoop:hadoop-ozone:[unknown-version],
> /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36,
> column 15
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194160#comment-16194160 ] Anu Engineer commented on HDFS-12608: - +1, pending jenkins. Thanks for fixing this. > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12608.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham reassigned HDFS-12608: - Assignee: Bharat Viswanadham > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12608.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194155#comment-16194155 ] Bharat Viswanadham commented on HDFS-12608: --- [~anu] Could you please review the changes. > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham > Attachments: HDFS-12608.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194153#comment-16194153 ] Bharat Viswanadham commented on HDFS-12608: --- Removed maven-project-info-reports-plugin, as this is removed from hadoop project. Refer https://issues.apache.org/jira/browse/HADOOP-14401. And the error is due to missing version tag for artifactId. > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham > Attachments: HDFS-12608.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12608: -- Attachment: HDFS-12608.00.patch > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham > Attachments: HDFS-12608.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12608: -- Status: Patch Available (was: Open) > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham > Attachments: HDFS-12608.00.patch > > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12520) Ozone : Add an API to get Open Container by Owner, Replication Type and Replication Count
[ https://issues.apache.org/jira/browse/HDFS-12520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-12520: Assignee: Lokesh Jain > Ozone : Add an API to get Open Container by Owner, Replication Type and > Replication Count > - > > Key: HDFS-12520 > URL: https://issues.apache.org/jira/browse/HDFS-12520 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Lokesh Jain > > During the code review of HDFS-12387 [~xyao] mentioned that it is a good idea > to have this API. This JIRA tracks that issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12521) Ozone: SCM should read all Container info into memory when booting up
[ https://issues.apache.org/jira/browse/HDFS-12521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh reassigned HDFS-12521: Assignee: Lokesh Jain > Ozone: SCM should read all Container info into memory when booting up > - > > Key: HDFS-12521 > URL: https://issues.apache.org/jira/browse/HDFS-12521 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Anu Engineer >Assignee: Lokesh Jain > Labels: performance > > When SCM boots up it should read all containers into memory. This is a > performance optimization that allows delays on SCM side. This JIRA tracks > that issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12602) Add Ozone config UI to KSM and SCM index page
[ https://issues.apache.org/jira/browse/HDFS-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12602: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~ajayydv] Thank you for the contribution. I have committed this to the feature branch. > Add Ozone config UI to KSM and SCM index page > -- > > Key: HDFS-12602 > URL: https://issues.apache.org/jira/browse/HDFS-12602 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Fix For: HDFS-7240 > > Attachments: HDFS-12602-HDFS-7240.01.patch > > > Add Ozone config UI to KSM and SCM index page -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Moved] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham moved HADOOP-14934 to HDFS-12608: Key: HDFS-12608 (was: HADOOP-14934) Project: Hadoop HDFS (was: Hadoop Common) > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12608) Remove Warnings when building ozone
[ https://issues.apache.org/jira/browse/HDFS-12608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12608: -- Issue Type: Sub-task (was: Bug) Parent: HDFS-7240 > Remove Warnings when building ozone > --- > > Key: HDFS-12608 > URL: https://issues.apache.org/jira/browse/HDFS-12608 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham > > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-ozone:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-project-info-reports-plugin is missing. @ > org.apache.hadoop:hadoop-ozone:[unknown-version], > /Users/aengineer/codereview/hadoop-tools/hadoop-ozone/pom.xml, line 36, > column 15 > [WARNING] > [WARNING] Some problems were encountered while building the effective model > for org.apache.hadoop:hadoop-dist:jar:3.1.0-SNAPSHOT > [WARNING] 'build.plugins.plugin.version' for > org.apache.maven.plugins:maven-gpg-plugin is missing. @ line 133, column 15 > [WARNING] > [WARNING] It is highly recommended to fix these problems because they > threaten the stability of your build. > [WARNING] > [WARNING] For this reason, future Maven versions might no longer support > building such malformed projects. > [WARNING] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12602) Add Ozone config UI to KSM and SCM index page
[
https://issues.apache.org/jira/browse/HDFS-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194140#comment-16194140
]
Hadoop QA commented on HDFS-12602:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
8s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12602 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890620/HDFS-12602-HDFS-7240.01.patch
|
| Optional Tests | asflicense shadedclient |
| uname | Linux 98cd7cf77380 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 5cb060a |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21557/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add Ozone config UI to KSM and SCM index page
> --
>
> Key: HDFS-12602
> URL: https://issues.apache.org/jira/browse/HDFS-12602
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: HDFS-7240
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: HDFS-7240
>
> Attachments: HDFS-12602-HDFS-7240.01.patch
>
>
> Add Ozone config UI to KSM and SCM index page
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12542) Update javadoc and documentation for listStatus
[ https://issues.apache.org/jira/browse/HDFS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12542: -- Status: Patch Available (was: Open) > Update javadoc and documentation for listStatus > > > Key: HDFS-12542 > URL: https://issues.apache.org/jira/browse/HDFS-12542 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12542.01.patch > > > Follow up jira to update javadoc and documentation for listStatus. > [HDFS-12162|https://issues.apache.org/jira/browse/HDFS-12162?focusedCommentId=16130910=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16130910] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12542) Update javadoc and documentation for listStatus
[ https://issues.apache.org/jira/browse/HDFS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12542: -- Attachment: (was: HDFS-12542.01.patch) > Update javadoc and documentation for listStatus > > > Key: HDFS-12542 > URL: https://issues.apache.org/jira/browse/HDFS-12542 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12542.01.patch > > > Follow up jira to update javadoc and documentation for listStatus. > [HDFS-12162|https://issues.apache.org/jira/browse/HDFS-12162?focusedCommentId=16130910=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16130910] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12542) Update javadoc and documentation for listStatus
[ https://issues.apache.org/jira/browse/HDFS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12542: -- Attachment: HDFS-12542.01.patch > Update javadoc and documentation for listStatus > > > Key: HDFS-12542 > URL: https://issues.apache.org/jira/browse/HDFS-12542 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12542.01.patch > > > Follow up jira to update javadoc and documentation for listStatus. > [HDFS-12162|https://issues.apache.org/jira/browse/HDFS-12162?focusedCommentId=16130910=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16130910] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12542) Update javadoc and documentation for listStatus
[ https://issues.apache.org/jira/browse/HDFS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12542: -- Attachment: HDFS-12542.01.patch > Update javadoc and documentation for listStatus > > > Key: HDFS-12542 > URL: https://issues.apache.org/jira/browse/HDFS-12542 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12542.01.patch > > > Follow up jira to update javadoc and documentation for listStatus. > [HDFS-12162|https://issues.apache.org/jira/browse/HDFS-12162?focusedCommentId=16130910=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16130910] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12542) Update javadoc and documentation for listStatus
[ https://issues.apache.org/jira/browse/HDFS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12542: -- Attachment: (was: HDFS-12542.01.patch) > Update javadoc and documentation for listStatus > > > Key: HDFS-12542 > URL: https://issues.apache.org/jira/browse/HDFS-12542 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar > > Follow up jira to update javadoc and documentation for listStatus. > [HDFS-12162|https://issues.apache.org/jira/browse/HDFS-12162?focusedCommentId=16130910=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16130910] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12542) Update javadoc and documentation for listStatus
[ https://issues.apache.org/jira/browse/HDFS-12542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12542: -- Attachment: HDFS-12542.01.patch > Update javadoc and documentation for listStatus > > > Key: HDFS-12542 > URL: https://issues.apache.org/jira/browse/HDFS-12542 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Attachments: HDFS-12542.01.patch > > > Follow up jira to update javadoc and documentation for listStatus. > [HDFS-12162|https://issues.apache.org/jira/browse/HDFS-12162?focusedCommentId=16130910=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16130910] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12570) [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements
[
https://issues.apache.org/jira/browse/HDFS-12570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194124#comment-16194124
]
Rakesh R edited comment on HDFS-12570 at 10/6/17 4:35 AM:
--
Thanks [~surendrasingh] for the reviews.
Attached latest patch fixing checkstyle, findbug, test case failure and
addressed (1) and (2) comments of Surendra.
comment-1) => Actually {{attemptedItemInfo.allBlockLocsAttemptedToSatisfy}}
attribute is not required as SPS will re-check the trackId/fileId once it
receives movementreport from DN. So, I've removed this flag.
comment-2) => Done
comment-3) => Interesting thought. In my current patch, kept blk movement
status reporting simple via heartbeat. As we know, IBR is reporting block
received and deleted items only. We need to change the semantics of
#blockReceivedAndDeleted call to incorporate the block move finished status.
Let me think for a while and get back to this point.
was (Author: rakeshr):
Thanks [~surendrasingh] for the reviews.
Attached latest patch fixing checkstyle, findbug, test case failure and
addressed (1) and (2) comments of Surendra.
comment-1) => Actually {{attemptedItemInfo.allBlockLocsAttemptedToSatisfy }}
attribute is not required as SPS will re-check the trackId/fileId once it
receives movementreport from DN. So, I've removed this flag.
comment-2) => Done
comment-3) => Interesting thought. In my current patch, kept blk movement
status reporting simple via heartbeat. As we know, IBR is reporting block
received and deleted items only. We need to change the semantics of
#blockReceivedAndDeleted call to incorporate the block move finished status.
Let me think for a while and get back to this point.
> [SPS]: Refactor Co-ordinator datanode logic to track the block storage
> movements
>
>
> Key: HDFS-12570
> URL: https://issues.apache.org/jira/browse/HDFS-12570
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12570-HDFS-10285-00.patch,
> HDFS-12570-HDFS-10285-01.patch
>
>
> This task is to refactor the C-DN block storage movements. Basically, the
> idea is to move the scheduling and tracking logic to Namenode rather than at
> the special C-DN. Please refer the discussion with [~andrew.wang] to
> understand the [background and the necessity of
> refactoring|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16141060=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16141060].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12570) [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements
[
https://issues.apache.org/jira/browse/HDFS-12570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194124#comment-16194124
]
Rakesh R commented on HDFS-12570:
-
Thanks [~surendrasingh] for the reviews.
Attached latest patch fixing checkstyle, findbug, test case failure and
addressed (1) and (2) comments of Surendra.
comment-1) => Actually {{attemptedItemInfo.allBlockLocsAttemptedToSatisfy }}
attribute is not required as SPS will re-check the trackId/fileId once it
receives movementreport from DN. So, I've removed this flag.
comment-2) => Done
comment-3) => Interesting thought. In my current patch, kept blk movement
status reporting simple via heartbeat. As we know, IBR is reporting block
received and deleted items only. We need to change the semantics of
#blockReceivedAndDeleted call to incorporate the block move finished status.
Let me think for a while and get back to this point.
> [SPS]: Refactor Co-ordinator datanode logic to track the block storage
> movements
>
>
> Key: HDFS-12570
> URL: https://issues.apache.org/jira/browse/HDFS-12570
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, namenode
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-12570-HDFS-10285-00.patch,
> HDFS-12570-HDFS-10285-01.patch
>
>
> This task is to refactor the C-DN block storage movements. Basically, the
> idea is to move the scheduling and tracking logic to Namenode rather than at
> the special C-DN. Please refer the discussion with [~andrew.wang] to
> understand the [background and the necessity of
> refactoring|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16141060=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16141060].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12602) Add Ozone config UI to KSM and SCM index page
[ https://issues.apache.org/jira/browse/HDFS-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12602: -- Status: Patch Available (was: Open) [~anu],[~xyao], could you plz have a look when possible. > Add Ozone config UI to KSM and SCM index page > -- > > Key: HDFS-12602 > URL: https://issues.apache.org/jira/browse/HDFS-12602 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Fix For: HDFS-7240 > > Attachments: HDFS-12602-HDFS-7240.01.patch > > > Add Ozone config UI to KSM and SCM index page -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12598) Ozone: Fix 3 node ratis replication in Ozone
[ https://issues.apache.org/jira/browse/HDFS-12598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194114#comment-16194114 ] Tsz Wo Nicholas Sze commented on HDFS-12598: Thanks a lot for fixing the bug. Great work! Some comments/questions: - createPipeline should not be called in newXceiverClientRatis; otherwise, it won't work for multiple clients in different machines. - Do we need to change the timeout default in order to pass the tests? - Use RatisTestSuite in TestKeysRatis. It will simplify the code a lot. > Ozone: Fix 3 node ratis replication in Ozone > > > Key: HDFS-12598 > URL: https://issues.apache.org/jira/browse/HDFS-12598 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12598-HDFS-7240.001.patch, > HDFS-12598-HDFS-7240.002.patch > > > Enabling ratis 3 node replication on ozone currently fails because > XceiverServerRatis doesn't initialize correctly because the server id is not > initialized correctly. Uploading an initial patch for early feedback. Will > upload an updated patch after testing on the cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-12420) Add an option to disallow 'namenode format -force'
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajay Kumar updated HDFS-12420:
--
Comment: was deleted
(was: [~arpitagarwal] thanks for review and commit.
[~shahrs87],[~daryn],[~aw],[~vagarychen],[~anu],[~vinayrpet],[~kihwal] thanks
for review and valuable feedback.)
> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch,
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch,
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch,
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch,
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of
> Namenode in production cluster. If someone really wants to delete the
> complete fsImage, they can first delete the metadata dir and then run {code}
> hdfs namenode -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
[
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194110#comment-16194110
]
Hadoop QA commented on HDFS-12605:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-9806 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
1s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
37s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
23s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} hadoop-tools/hadoop-fs2img in HDFS-9806 has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} HDFS-9806 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}119m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
50s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}221m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestReplaceDatanodeFailureReplication |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12605 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890622/HDFS-12605-HDFS-9806.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4df4c67eb7ed 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (HDFS-12570) [SPS]: Refactor Co-ordinator datanode logic to track the block storage movements
[ https://issues.apache.org/jira/browse/HDFS-12570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh R updated HDFS-12570: Attachment: HDFS-12570-HDFS-10285-01.patch > [SPS]: Refactor Co-ordinator datanode logic to track the block storage > movements > > > Key: HDFS-12570 > URL: https://issues.apache.org/jira/browse/HDFS-12570 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R >Assignee: Rakesh R > Attachments: HDFS-12570-HDFS-10285-00.patch, > HDFS-12570-HDFS-10285-01.patch > > > This task is to refactor the C-DN block storage movements. Basically, the > idea is to move the scheduling and tracking logic to Namenode rather than at > the special C-DN. Please refer the discussion with [~andrew.wang] to > understand the [background and the necessity of > refactoring|https://issues.apache.org/jira/browse/HDFS-10285?focusedCommentId=16141060=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16141060]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194104#comment-16194104 ] Xiao Chen commented on HDFS-12603: -- seems infra, re-kicked jenkins > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12598) Ozone: Fix 3 node ratis replication in Ozone
[ https://issues.apache.org/jira/browse/HDFS-12598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh updated HDFS-12598: - Attachment: HDFS-12598-HDFS-7240.002.patch patch v2 fixes the build issue in the last patch > Ozone: Fix 3 node ratis replication in Ozone > > > Key: HDFS-12598 > URL: https://issues.apache.org/jira/browse/HDFS-12598 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12598-HDFS-7240.001.patch, > HDFS-12598-HDFS-7240.002.patch > > > Enabling ratis 3 node replication on ozone currently fails because > XceiverServerRatis doesn't initialize correctly because the server id is not > initialized correctly. Uploading an initial patch for early feedback. Will > upload an updated patch after testing on the cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12555) HDFS federation should support configure secondary directory
[ https://issues.apache.org/jira/browse/HDFS-12555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194103#comment-16194103 ] Bharat Viswanadham commented on HDFS-12555: --- [~luoge123] Could you please add UT's to test the scenario? > HDFS federation should support configure secondary directory > - > > Key: HDFS-12555 > URL: https://issues.apache.org/jira/browse/HDFS-12555 > Project: Hadoop HDFS > Issue Type: Improvement > Components: federation > Environment: 2.6.0-cdh5.10.0 >Reporter: luoge123 > Fix For: 2.6.0 > > Attachments: HDFS-12555.001.patch > > > HDFS federation support multiple namenodes horizontally scales the file > system namespace. As the amount of data grows, using a single group of > namenodes to manage a single directory, namenode still achieves performance > bottlenecks. In order to reduce the pressure of namenode, we can split out > the secondary directory, and manager it by a new namenode. This is > transparent for users. > For example, nn1 only manager the /user directory, when nn1 achieve > performance bottlenecks, we can split out /user/hive directory, and ues nn2 > to manager it. > That means core-site.xml should support as follows configuration. > >fs.viewfs.mounttable.nsX.link./user >hdfs://nn1:8020/user > > >fs.viewfs.mounttable.nsX.link./user/hive >hdfs://nn2:8020/user/hive > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12577) Rename Router tooling
[
https://issues.apache.org/jira/browse/HDFS-12577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194099#comment-16194099
]
Hadoop QA commented on HDFS-12577:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-10467 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
54s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} HDFS-10467 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} HDFS-10467 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
23s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 27s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}134m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12577 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890629/HDFS-12577-HDFS-10467-001.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs compile
javac javadoc mvninstall shadedclient findbugs checkstyle |
| uname | Linux b0e9b4ea5491 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-10467 / c0ad434 |
| Default Java | 1.8.0_144 |
| shellcheck | v0.4.6 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Updated] (HDFS-12593) Ozone: update Ratis to the latest snapshot
[ https://issues.apache.org/jira/browse/HDFS-12593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-12593: --- Attachment: HDFS-12593-HDFS-7240.20171006.patch [~linyiqun], thanks for checking the build. The version is incorrect in the previous patch. Here is a new patch. HDFS-12593-HDFS-7240.20171006.patch > Ozone: update Ratis to the latest snapshot > -- > > Key: HDFS-12593 > URL: https://issues.apache.org/jira/browse/HDFS-12593 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-12593-HDFS-7240.20171005.patch, > HDFS-12593-HDFS-7240.20171006.patch > > > Apache Ratis has quite a few bug fixes in the latest snapshot (7a5c3ea). Let > update to it. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12603) Enable async edit logging by default
[
https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194069#comment-16194069
]
Hadoop QA commented on HDFS-12603:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}210m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}300m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReservedRawPaths |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100 |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190 |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.TestEncryptionZones |
| |
[jira] [Comment Edited] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193915#comment-16193915 ] Bharat Viswanadham edited comment on HDFS-12582 at 10/6/17 3:10 AM: [~vagarychen] Updated the patch v02. Fixed UT failures and checkstyle issues. Test failures are due to error in patch v01, fixed it. Tested locally. Test cases are passing. was (Author: bharatviswa): [~vagarychen] Updated the patch v02. Fixed UT failures and checkstyle issues. Test failures is due to error in code, fixed it. Tested locally. Test cases are passing. > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch, > HDFS-12582.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12582: -- Attachment: HDFS-12582.03.patch > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch, > HDFS-12582.03.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[
https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194066#comment-16194066
]
Bharat Viswanadham commented on HDFS-12582:
---
Uploaded patch v03 to fix checkstyle issues.
Not resolved these checkstyle issues which are not from this change.
{noformat}
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java:84:
protected HdfsFileStatus(long length, boolean isdir, int
block_replication,:13: More than 7 parameters (found 17). [ParameterNumber]
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java:84:
protected HdfsFileStatus(long length, boolean isdir, int
block_replication,:60: Name 'block_replication' must match pattern
'^[a-z][a-zA-Z0-9]*$'. [ParameterName]
{noformat}
> Replace HdfsFileStatus constructor with a builder pattern.
> --
>
> Key: HDFS-12582
> URL: https://issues.apache.org/jira/browse/HDFS-12582
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
> Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch,
> HDFS-12582.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[
https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194066#comment-16194066
]
Bharat Viswanadham edited comment on HDFS-12582 at 10/6/17 3:06 AM:
Uploaded patch v03 to fix checkstyle issues.
Not resolved these checkstyle issues which are not from this change.
{code:java}
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java:84:
protected HdfsFileStatus(long length, boolean isdir, int
block_replication,:13: More than 7 parameters (found 17). [ParameterNumber]
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java:84:
protected HdfsFileStatus(long length, boolean isdir, int
block_replication,:60: Name 'block_replication' must match pattern
'^[a-z][a-zA-Z0-9]*$'. [ParameterName]
{code}
was (Author: bharatviswa):
Uploaded patch v03 to fix checkstyle issues.
Not resolved these checkstyle issues which are not from this change.
{noformat}
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java:84:
protected HdfsFileStatus(long length, boolean isdir, int
block_replication,:13: More than 7 parameters (found 17). [ParameterNumber]
./hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsFileStatus.java:84:
protected HdfsFileStatus(long length, boolean isdir, int
block_replication,:60: Name 'block_replication' must match pattern
'^[a-z][a-zA-Z0-9]*$'. [ParameterName]
{noformat}
> Replace HdfsFileStatus constructor with a builder pattern.
> --
>
> Key: HDFS-12582
> URL: https://issues.apache.org/jira/browse/HDFS-12582
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Bharat Viswanadham
>Assignee: Bharat Viswanadham
> Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch,
> HDFS-12582.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194061#comment-16194061 ] Bharat Viswanadham commented on HDFS-12582: --- Test failures are not related to this patch. Ran tests locally and tests have passed. > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12411) Ozone: Add container usage information to DN container report
[
https://issues.apache.org/jira/browse/HDFS-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194057#comment-16194057
]
Hadoop QA commented on HDFS-12411:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
59s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
55s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
0s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 4 new +
3 unchanged - 0 fixed = 7 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
31s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}136m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}207m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.cblock.TestBufferManager |
| | hadoop.ozone.ksm.TestMultipleContainerReadWrite |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.cblock.TestCBlockReadWrite |
| Timed out junit tests | org.apache.hadoop.cblock.TestLocalBlockCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12411 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[
https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194052#comment-16194052
]
Hadoop QA commented on HDFS-12582:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project: The patch generated 5 new +
290 unchanged - 3 fixed = 295 total (was 293) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 39s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 14s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12582 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890616/HDFS-12582.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5116d96aff53 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c071aad |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[
https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194019#comment-16194019
]
Hadoop QA commented on HDFS-12600:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 23s{color} | {color:orange} root: The patch generated 2 new + 772 unchanged
- 7 fixed = 774 total (was 779) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 59s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 9s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
20s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.net.TestDNS |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12600 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890600/HDFS-12600.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2b0aaa19cca0 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64
[jira] [Commented] (HDFS-9962) Erasure Coding: need a way to test multiple EC policies
[ https://issues.apache.org/jira/browse/HDFS-9962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194007#comment-16194007 ] Andrew Wang commented on HDFS-9962: --- Hi [~tasanuma0829], thanks for working on this, Is your concern about flakiness related to test timeouts? We've been struggling with that recently, since increasing the cell size from 64k to 1MB greatly increased test runtimes. The approach we've been taking is to split long running tests into separate classes, or "parameterizing" by creating a lot of subclasses. We randomized the policy tests to reduce the runtime of the test suite, under the belief that there wasn't much incremental benefit from running all of them every time. I think that's still true, and would prefer we randomize any long-running tests. If the tests run quickly, then it's okay to be exhaustive. > Erasure Coding: need a way to test multiple EC policies > --- > > Key: HDFS-9962 > URL: https://issues.apache.org/jira/browse/HDFS-9962 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Rui Li >Assignee: Takanobu Asanuma > Labels: hdfs-ec-3.0-nice-to-have > > Now that we support multiple EC policies, we need a way test it to catch > potential issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12577) Rename Router tooling
[
https://issues.apache.org/jira/browse/HDFS-12577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-12577:
---
Attachment: HDFS-12577-HDFS-10467-001.patch
Fixed checkstyle but not sure we should.
> Rename Router tooling
> -
>
> Key: HDFS-12577
> URL: https://issues.apache.org/jira/browse/HDFS-12577
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
> Labels: RBF
> Fix For: HDFS-10467
>
> Attachments: HDFS-12577-HDFS-10467-000.patch,
> HDFS-12577-HDFS-10467-001.patch
>
>
> Currently the naming for Router Based Federation has a couple conflicts:
> * Both YARN and HDFS have a Router component which may cause issues for the
> PID file and JPS.
> * The tool to manage the mount table is called using {{hdfs federation}}.
> This may cause confusion with the regular HDFS federation.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Attachment: HDFS-12607-HDFS-9806.001.patch
Patch fixes the issue by changing the state of the ProvidedStorageInfo to
FAILED only if all DNs fail, and restoring the state to NORMAL when a DN with
Provided storage reports.
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607-HDFS-9806.001.patch, HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode to NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12606) JVM crashes when running NNBench on EC enabled.
[
https://issues.apache.org/jira/browse/HDFS-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193998#comment-16193998
]
Kai Zheng commented on HDFS-12606:
--
Thanks for the ping Eddy. By design we can have multiple coder instances for
concurrent coding tasks, and no global static variable should block this except
bugs. We guard isal codes in Java, not relying on its thread model. We can
investigate it when back to office, next Monday.
> JVM crashes when running NNBench on EC enabled.
>
>
> Key: HDFS-12606
> URL: https://issues.apache.org/jira/browse/HDFS-12606
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Priority: Critical
>
> When running NNbench on a RS(6,3) directory, JVM crashes double free or
> corruption:
> {code}
> 08:16:29 Running NNBENCH.
> 08:16:29 WARNING: Use "yarn jar" to launch YARN applications.
> 08:16:31 NameNode Benchmark 0.4
> 08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Test Inputs:
> 08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Test Operation: create_write
> 08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Start time: 2017-10-04
> 08:18:31,16
> :
> :
> 08:18:54 *** Error in `/usr/java/jdk1.8.0_144/bin/java': double free or
> corruption (out): 0x7ffb55dbfab0 ***
> 08:18:54 === Backtrace: =
> 08:18:54 /lib64/libc.so.6(+0x7c619)[0x7ffb5b85f619]
> 08:18:54 [0x7ffb45017774]
> 08:18:54 === Memory map:
> 08:18:54 0040-00401000 r-xp ca:01 276832134
> /usr/java/jdk1.8.0_144/bin/java
> 08:18:54 0060-00601000 rw-p ca:01 276832134
> /usr/java/jdk1.8.0_144/bin/java
> 08:18:54 0173e000-01f91000 rw-p 00:00 0 [heap]
> 08:18:54 60360-61470 rw-p 00:00 0
> 08:18:54 61470-72bd0 ---p 00:00 0
> 08:18:54 72bd0-73a50 rw-p 00:00 0
> 08:18:54 73a50-7c000 ---p 00:00 0
> 08:18:54 7c000-7c040 rw-p 00:00 0
> 08:18:54 7c040-8 ---p 00:00 0
> 08:18:54 7ffb20174000-7ffb208ab000 rw-p 00:00 0
> 08:18:54 7ffb208ab000-7ffb20975000 ---p 00:00 0
> 08:18:54 7ffb20975000-7ffb20b75000 rw-p 00:00 0
> 08:18:54 7ffb20b75000-7ffb20d75000 rw-p 00:00 0
> 08:18:54 7ffb20d75000-7ffb20d8a000 r-xp ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20d8a000-7ffb20f89000 ---p 00015000 ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20f89000-7ffb20f8a000 r--p 00014000 ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20f8a000-7ffb20f8b000 rw-p 00015000 ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20f8b000-7ffb20fbd000 r-xp ca:01 553654092
> /usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
> 08:18:54 7ffb20fbd000-7ffb211bc000 ---p 00032000 ca:01 553654092
> /usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
> 08:18:54 7ffb211bc000-7ffb211c2000 rw-p 00031000 ca:01 553654092
> /usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
> :
> :
> 08:18:54 7ffb5c3fb000-7ffb5c3fc000 r--p 00:00 0
> 08:18:54 7ffb5c3fc000-7ffb5c3fd000 rw-p 00:00 0
> 08:18:54 7ffb5c3fd000-7ffb5c3fe000 r--p 00021000 ca:01 637266
> /usr/lib64/ld-2.17.so
> 08:18:54 7ffb5c3fe000-7ffb5c3ff000 rw-p 00022000 ca:01 637266
> /usr/lib64/ld-2.17.so
> 08:18:54 7ffb5c3ff000-7ffb5c40 rw-p 00:00 0
> 08:18:54 7ffdf8767000-7ffdf8788000 rw-p 00:00 0 [stack]
> 08:18:54 7ffdf878b000-7ffdf878d000 r-xp 00:00 0 [vdso]
> 08:18:54 ff60-ff601000 r-xp 00:00 0 [vsyscall]
> {code}
> It happens on both {{jdk1.8.0_144}} and {{jdk1.8.0_121}} in our environments.
> It is highly suspicious due to the native code used in erasure coding, i.e.,
> ISA-L is not thread safe
> [https://01.org/sites/default/files/documentation/isa-l_open_src_2.10.pdf]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12575) Improve test coverage for EC related edit logs ops
[
https://issues.apache.org/jira/browse/HDFS-12575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-12575:
---
Labels: hdfs-ec-3.0-nice-to-have (was: )
> Improve test coverage for EC related edit logs ops
> --
>
> Key: HDFS-12575
> URL: https://issues.apache.org/jira/browse/HDFS-12575
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha4
>Reporter: Lei (Eddy) Xu
>Assignee: SammiChen
> Labels: hdfs-ec-3.0-nice-to-have
>
> HDFS-12569 found that we have little test coverage for edit logs ops of
> erasure coding.
> And we've seen the following bug bring down SNN in our test environments:
> {code}
> 6:42:18.177 AMERROR FSEditLogLoader
> Encountered exception on operation AddBlockOp [path=/tmp/foo/bar,
> penultimateBlock=NULL, lastBlock=blk_1073743386_69322, RpcClientId=,
> RpcCallId=-2]
> java.lang.IllegalArgumentException: reportedBlock is not striped
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
>
> 6:42:18.190 AMFATAL EditLogTailer
> Unknown error encountered while tailing edits. Shutting down standby NN.
> java.io.IOException: java.lang.IllegalArgumentException: reportedBlock is not
> striped
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:251)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:150)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:882)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:863)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:293)
> at
> {code}
> We should add coverage for these important edit logs, i.e., set/unset policy,
> enable/remove policies and etc are correctly persisted in edit logs, and test
> the scenarios like:
> * Restart NN
> * Replay edits after checkpoint
> * Apply edits on SNN.
> * and etc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Attachment: HDFS-12607.repro.patch
Attaching patch with a test case to reproduce the issue.
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12607.repro.patch
>
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode to NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[
https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193983#comment-16193983
]
Hadoop QA commented on HDFS-12543:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 5s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
41s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 15s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}155m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.ozone.scm.TestContainerSQLCli |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12543 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-10467) Router-based HDFS federation
[ https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193982#comment-16193982 ] Íñigo Goiri commented on HDFS-10467: The [vote for merging|http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201709.mbox/%3CCAB1dGgoyfOM5ydbHrFQSbxq9Q3Kt-B1Xz3qFiXAe%2BNWyrbc3-Q%40mail.gmail.com%3E] is finishing tomorrow. At this point, the only thing open is the issue with the names described in HDFS-12577. The security patch can be moved to v2 (I'll open a new umbrella with the remaining issues). > Router-based HDFS federation > > > Key: HDFS-10467 > URL: https://issues.apache.org/jira/browse/HDFS-10467 > Project: Hadoop HDFS > Issue Type: New Feature > Components: fs >Affects Versions: 2.8.1 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Labels: RBF > Fix For: HDFS-10467 > > Attachments: HDFS-10467.002.patch, HDFS-10467.PoC.001.patch, > HDFS-10467.PoC.patch, HDFS Router Federation.pdf, > HDFS-Router-Federation-Prototype.patch > > > Add a Router to provide a federated view of multiple HDFS clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193978#comment-16193978 ] Íñigo Goiri commented on HDFS-12273: I just committed this. [~giovanni.fumarola] and [~chris.douglas], thanks for the comments and the review! > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: federationUI-1.png, federationUI-2.png, > federationUI-3.png, HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch, HDFS-12273-HDFS-10467-002.patch, > HDFS-12273-HDFS-10467-003.patch, HDFS-12273-HDFS-10467-004.patch, > HDFS-12273-HDFS-10467-005.patch, HDFS-12273-HDFS-10467-006.patch, > HDFS-12273-HDFS-10467-007.patch, HDFS-12273-HDFS-10467-008.patch, > HDFS-12273-HDFS-10467-009.patch, HDFS-12273-HDFS-10467-010.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
[
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12605:
--
Description:
{{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails after
rebase with the following error:
{code}
java.lang.NullPointerException: null
at
org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
at
org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
at
org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
at
org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
at
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
at
org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
at
org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
{code}
was:{{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails
after rebase.
> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
> at
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12567) BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes
[
https://issues.apache.org/jira/browse/HDFS-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193968#comment-16193968
]
Hudson commented on HDFS-12567:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13038 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13038/])
HDFS-12567. BlockPlacementPolicyRackFaultTolerant fails with racks with (wang:
rev 644c2f6924f341f51d809c91dccfff88fc82f6f0)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingMultipleRacks.java
> BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes
> --
>
> Key: HDFS-12567
> URL: https://issues.apache.org/jira/browse/HDFS-12567
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0
>
> Attachments: HDFS-12567.001.patch, HDFS-12567.002.patch,
> HDFS-12567.003.patch, HDFS-12567.repro.patch
>
>
> Found this while doing some testing on an internal cluster with an unusual
> setup. We have a rack with ~20 nodes, then a few more with just a few nodes.
> It would fail to get (# data blocks) datanodes even though there were plenty
> of DNs on the rack with 20 DNs.
> I managed to reproduce this same issue in a unit test, stack trace like this:
> {noformat}
> java.io.IOException: File /testfile0 could only be written to 5 of the 6
> required nodes for RS-6-3-1024k. There are 9 datanode(s) running and no
> node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2083)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:286)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2609)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:863)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:548)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {noformat}
> This isn't a very critical bug since it's an unusual rack configuration, but
> it can easily happen during testing.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-12273: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: federationUI-1.png, federationUI-2.png, > federationUI-3.png, HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch, HDFS-12273-HDFS-10467-002.patch, > HDFS-12273-HDFS-10467-003.patch, HDFS-12273-HDFS-10467-004.patch, > HDFS-12273-HDFS-10467-005.patch, HDFS-12273-HDFS-10467-006.patch, > HDFS-12273-HDFS-10467-007.patch, HDFS-12273-HDFS-10467-008.patch, > HDFS-12273-HDFS-10467-009.patch, HDFS-12273-HDFS-10467-010.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193959#comment-16193959 ] Íñigo Goiri commented on HDFS-12273: Given that the 1 second time out is mainly for the UI, we could increase it a little but not much more. > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: federationUI-1.png, federationUI-2.png, > federationUI-3.png, HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch, HDFS-12273-HDFS-10467-002.patch, > HDFS-12273-HDFS-10467-003.patch, HDFS-12273-HDFS-10467-004.patch, > HDFS-12273-HDFS-10467-005.patch, HDFS-12273-HDFS-10467-006.patch, > HDFS-12273-HDFS-10467-007.patch, HDFS-12273-HDFS-10467-008.patch, > HDFS-12273-HDFS-10467-009.patch, HDFS-12273-HDFS-10467-010.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
[
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193961#comment-16193961
]
Virajith Jalaparti edited comment on HDFS-12605 at 10/6/17 12:26 AM:
-
The patch fixes the failing test case by adding the PROVIDED storage type to
the {{DFSTopologyNodeImpl}}.
One thing I noticed in the test case was that the 2nd DN stopped was different
from the datanode passed to {{BlockManagerTestUtil.noticeDeadDatanode}}. This
lead to the NN thinking the DN was dead even though the DN was alive and
continues to heart beat to the NN. As a result, the ProvidedStorageInfo was
marked as FAILED, and is never set back to NORMAL again. This issue will be
fixed in HDFS-12607.
was (Author: virajith):
The patch fixes the failing test case by adding the PROVIDED storage type to
the {{DFSTopologyNodeImpl}}
> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
> at
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12607) [READ] Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
[
https://issues.apache.org/jira/browse/HDFS-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12607:
--
Summary: [READ] Even one dead datanode with PROVIDED storage results in
ProvidedStorageInfo being marked as FAILED (was: Even one dead datanode with
PROVIDED storage results in ProvidedStorageInfo being marked as FAILED)
> [READ] Even one dead datanode with PROVIDED storage results in
> ProvidedStorageInfo being marked as FAILED
> -
>
> Key: HDFS-12607
> URL: https://issues.apache.org/jira/browse/HDFS-12607
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>
> When a DN configured with PROVIDED storage is marked as dead by the NN, the
> state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
> and never becomes NORMAL. The state should change to FAILED only if all
> datanodes with PROVIDED storage are dead, and should be restored back to
> NORMAL when a Datanode to NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
[
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12605:
--
Attachment: HDFS-12605-HDFS-9806.001.patch
The patch fixes the failing test case by adding the PROVIDED storage type to
the {{DFSTopologyNodeImpl}}
> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
> at
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12607) Even one dead datanode with PROVIDED storage results in ProvidedStorageInfo being marked as FAILED
Virajith Jalaparti created HDFS-12607:
-
Summary: Even one dead datanode with PROVIDED storage results in
ProvidedStorageInfo being marked as FAILED
Key: HDFS-12607
URL: https://issues.apache.org/jira/browse/HDFS-12607
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Virajith Jalaparti
Assignee: Virajith Jalaparti
When a DN configured with PROVIDED storage is marked as dead by the NN, the
state of {{providedStorageInfo}} in {{ProvidedStorageMap}} is set to FAILED,
and never becomes NORMAL. The state should change to FAILED only if all
datanodes with PROVIDED storage are dead, and should be restored back to NORMAL
when a Datanode to NORMAL DatanodeStorage reports in.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12599) Move DataNodeTestUtils.mockDatanodeBlkPinning into mock test util class
[
https://issues.apache.org/jira/browse/HDFS-12599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193960#comment-16193960
]
Ted Yu commented on HDFS-12599:
---
Don't think the failed tests were related to the patch.
Ran TestDFSStripedOutputStream\* locally which passed.
{code}
Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
Tests run: 16, Failures: 0, Errors: 0, Skipped: 12, Time elapsed: 497.37 sec -
in org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
{code}
The above test needs some speedup.
> Move DataNodeTestUtils.mockDatanodeBlkPinning into mock test util class
> ---
>
> Key: HDFS-12599
> URL: https://issues.apache.org/jira/browse/HDFS-12599
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0-beta1
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: HDFS-12599.v1.patch
>
>
> HDFS-11164 introduced {{DataNodeTestUtils.mockDatanodeBlkPinning}} which
> brought dependency on mockito back into DataNodeTestUtils
> Downstream, this resulted in:
> {code}
> java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.shouldWait(MiniDFSCluster.java:2668)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2564)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.waitActive(MiniDFSCluster.java:2607)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1667)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:874)
> at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:769)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:661)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1075)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:953)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12606) JVM crashes when running NNBench on EC enabled.
[
https://issues.apache.org/jira/browse/HDFS-12606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193973#comment-16193973
]
Lei (Eddy) Xu commented on HDFS-12606:
--
I saw a single static global variable {{isaLoader}} in {{isal_loader.h}}
providing access to {{ISA-L}} library. One simple way is to make
{{NativeRSRawDecoder}}, but it will limits the parallelism of encoding /
decoding on DN. [~Kai], [~Sammi] Could you give some thoughts about this?
> JVM crashes when running NNBench on EC enabled.
>
>
> Key: HDFS-12606
> URL: https://issues.apache.org/jira/browse/HDFS-12606
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Priority: Critical
>
> When running NNbench on a RS(6,3) directory, JVM crashes double free or
> corruption:
> {code}
> 08:16:29 Running NNBENCH.
> 08:16:29 WARNING: Use "yarn jar" to launch YARN applications.
> 08:16:31 NameNode Benchmark 0.4
> 08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Test Inputs:
> 08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Test Operation: create_write
> 08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Start time: 2017-10-04
> 08:18:31,16
> :
> :
> 08:18:54 *** Error in `/usr/java/jdk1.8.0_144/bin/java': double free or
> corruption (out): 0x7ffb55dbfab0 ***
> 08:18:54 === Backtrace: =
> 08:18:54 /lib64/libc.so.6(+0x7c619)[0x7ffb5b85f619]
> 08:18:54 [0x7ffb45017774]
> 08:18:54 === Memory map:
> 08:18:54 0040-00401000 r-xp ca:01 276832134
> /usr/java/jdk1.8.0_144/bin/java
> 08:18:54 0060-00601000 rw-p ca:01 276832134
> /usr/java/jdk1.8.0_144/bin/java
> 08:18:54 0173e000-01f91000 rw-p 00:00 0 [heap]
> 08:18:54 60360-61470 rw-p 00:00 0
> 08:18:54 61470-72bd0 ---p 00:00 0
> 08:18:54 72bd0-73a50 rw-p 00:00 0
> 08:18:54 73a50-7c000 ---p 00:00 0
> 08:18:54 7c000-7c040 rw-p 00:00 0
> 08:18:54 7c040-8 ---p 00:00 0
> 08:18:54 7ffb20174000-7ffb208ab000 rw-p 00:00 0
> 08:18:54 7ffb208ab000-7ffb20975000 ---p 00:00 0
> 08:18:54 7ffb20975000-7ffb20b75000 rw-p 00:00 0
> 08:18:54 7ffb20b75000-7ffb20d75000 rw-p 00:00 0
> 08:18:54 7ffb20d75000-7ffb20d8a000 r-xp ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20d8a000-7ffb20f89000 ---p 00015000 ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20f89000-7ffb20f8a000 r--p 00014000 ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20f8a000-7ffb20f8b000 rw-p 00015000 ca:01 209866
> /usr/lib64/libgcc_s-4.8.5-20150702.so.1
> 08:18:54 7ffb20f8b000-7ffb20fbd000 r-xp ca:01 553654092
> /usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
> 08:18:54 7ffb20fbd000-7ffb211bc000 ---p 00032000 ca:01 553654092
> /usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
> 08:18:54 7ffb211bc000-7ffb211c2000 rw-p 00031000 ca:01 553654092
> /usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
> :
> :
> 08:18:54 7ffb5c3fb000-7ffb5c3fc000 r--p 00:00 0
> 08:18:54 7ffb5c3fc000-7ffb5c3fd000 rw-p 00:00 0
> 08:18:54 7ffb5c3fd000-7ffb5c3fe000 r--p 00021000 ca:01 637266
> /usr/lib64/ld-2.17.so
> 08:18:54 7ffb5c3fe000-7ffb5c3ff000 rw-p 00022000 ca:01 637266
> /usr/lib64/ld-2.17.so
> 08:18:54 7ffb5c3ff000-7ffb5c40 rw-p 00:00 0
> 08:18:54 7ffdf8767000-7ffdf8788000 rw-p 00:00 0 [stack]
> 08:18:54 7ffdf878b000-7ffdf878d000 r-xp 00:00 0 [vdso]
> 08:18:54 ff60-ff601000 r-xp 00:00 0 [vsyscall]
> {code}
> It happens on both {{jdk1.8.0_144}} and {{jdk1.8.0_121}} in our environments.
> It is highly suspicious due to the native code used in erasure coding, i.e.,
> ISA-L is not thread safe
> [https://01.org/sites/default/files/documentation/isa-l_open_src_2.10.pdf]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12606) JVM crashes when running NNBench on EC enabled.
Lei (Eddy) Xu created HDFS-12606:
Summary: JVM crashes when running NNBench on EC enabled.
Key: HDFS-12606
URL: https://issues.apache.org/jira/browse/HDFS-12606
Project: Hadoop HDFS
Issue Type: Bug
Components: erasure-coding
Affects Versions: 3.0.0-beta1
Reporter: Lei (Eddy) Xu
Priority: Critical
When running NNbench on a RS(6,3) directory, JVM crashes double free or
corruption:
{code}
08:16:29 Running NNBENCH.
08:16:29 WARNING: Use "yarn jar" to launch YARN applications.
08:16:31 NameNode Benchmark 0.4
08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Test Inputs:
08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Test Operation: create_write
08:16:31 17/10/04 08:16:31 INFO hdfs.NNBench: Start time: 2017-10-04 08:18:31,16
:
:
08:18:54 *** Error in `/usr/java/jdk1.8.0_144/bin/java': double free or
corruption (out): 0x7ffb55dbfab0 ***
08:18:54 === Backtrace: =
08:18:54 /lib64/libc.so.6(+0x7c619)[0x7ffb5b85f619]
08:18:54 [0x7ffb45017774]
08:18:54 === Memory map:
08:18:54 0040-00401000 r-xp ca:01 276832134
/usr/java/jdk1.8.0_144/bin/java
08:18:54 0060-00601000 rw-p ca:01 276832134
/usr/java/jdk1.8.0_144/bin/java
08:18:54 0173e000-01f91000 rw-p 00:00 0 [heap]
08:18:54 60360-61470 rw-p 00:00 0
08:18:54 61470-72bd0 ---p 00:00 0
08:18:54 72bd0-73a50 rw-p 00:00 0
08:18:54 73a50-7c000 ---p 00:00 0
08:18:54 7c000-7c040 rw-p 00:00 0
08:18:54 7c040-8 ---p 00:00 0
08:18:54 7ffb20174000-7ffb208ab000 rw-p 00:00 0
08:18:54 7ffb208ab000-7ffb20975000 ---p 00:00 0
08:18:54 7ffb20975000-7ffb20b75000 rw-p 00:00 0
08:18:54 7ffb20b75000-7ffb20d75000 rw-p 00:00 0
08:18:54 7ffb20d75000-7ffb20d8a000 r-xp ca:01 209866
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
08:18:54 7ffb20d8a000-7ffb20f89000 ---p 00015000 ca:01 209866
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
08:18:54 7ffb20f89000-7ffb20f8a000 r--p 00014000 ca:01 209866
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
08:18:54 7ffb20f8a000-7ffb20f8b000 rw-p 00015000 ca:01 209866
/usr/lib64/libgcc_s-4.8.5-20150702.so.1
08:18:54 7ffb20f8b000-7ffb20fbd000 r-xp ca:01 553654092
/usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
08:18:54 7ffb20fbd000-7ffb211bc000 ---p 00032000 ca:01 553654092
/usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
08:18:54 7ffb211bc000-7ffb211c2000 rw-p 00031000 ca:01 553654092
/usr/java/jdk1.8.0_144/jre/lib/amd64/libsunec.so
:
:
08:18:54 7ffb5c3fb000-7ffb5c3fc000 r--p 00:00 0
08:18:54 7ffb5c3fc000-7ffb5c3fd000 rw-p 00:00 0
08:18:54 7ffb5c3fd000-7ffb5c3fe000 r--p 00021000 ca:01 637266
/usr/lib64/ld-2.17.so
08:18:54 7ffb5c3fe000-7ffb5c3ff000 rw-p 00022000 ca:01 637266
/usr/lib64/ld-2.17.so
08:18:54 7ffb5c3ff000-7ffb5c40 rw-p 00:00 0
08:18:54 7ffdf8767000-7ffdf8788000 rw-p 00:00 0 [stack]
08:18:54 7ffdf878b000-7ffdf878d000 r-xp 00:00 0 [vdso]
08:18:54 ff60-ff601000 r-xp 00:00 0 [vsyscall]
{code}
It happens on both {{jdk1.8.0_144}} and {{jdk1.8.0_121}} in our environments.
It is highly suspicious due to the native code used in erasure coding, i.e.,
ISA-L is not thread safe
[https://01.org/sites/default/files/documentation/isa-l_open_src_2.10.pdf]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
[
https://issues.apache.org/jira/browse/HDFS-12605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Virajith Jalaparti updated HDFS-12605:
--
Status: Patch Available (was: Open)
> [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails
> after rebase
> -
>
> Key: HDFS-12605
> URL: https://issues.apache.org/jira/browse/HDFS-12605
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-12605-HDFS-9806.001.patch
>
>
> {{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails
> after rebase with the following error:
> {code}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.decStorageTypeCount(DFSTopologyNodeImpl.java:127)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:318)
> at
> org.apache.hadoop.hdfs.net.DFSTopologyNodeImpl.remove(DFSTopologyNodeImpl.java:336)
> at
> org.apache.hadoop.net.NetworkTopology.remove(NetworkTopology.java:222)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDatanode(DatanodeManager.java:712)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.removeDeadDatanode(DatanodeManager.java:755)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager.heartbeatCheck(HeartbeatManager.java:407)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil.noticeDeadDatanode(BlockManagerTestUtil.java:213)
> at
> org.apache.hadoop.hdfs.server.namenode.TestNameNodeProvidedImplementation.testProvidedDatanodeFailures(TestNameNodeProvidedImplementation.java:471)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12605) [READ] TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after rebase
Virajith Jalaparti created HDFS-12605:
-
Summary: [READ]
TestNameNodeProvidedImplementation#testProvidedDatanodeFailures fails after
rebase
Key: HDFS-12605
URL: https://issues.apache.org/jira/browse/HDFS-12605
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Virajith Jalaparti
Assignee: Virajith Jalaparti
{{TestNameNodeProvidedImplementation#testProvidedDatanodeFailures}} fails after
rebase.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12602) Add Ozone config UI to KSM and SCM index page
[ https://issues.apache.org/jira/browse/HDFS-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDFS-12602: -- Attachment: HDFS-12602-HDFS-7240.01.patch > Add Ozone config UI to KSM and SCM index page > -- > > Key: HDFS-12602 > URL: https://issues.apache.org/jira/browse/HDFS-12602 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: HDFS-7240 >Reporter: Ajay Kumar >Assignee: Ajay Kumar > Fix For: HDFS-7240 > > Attachments: HDFS-12602-HDFS-7240.01.patch > > > Add Ozone config UI to KSM and SCM index page -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Add an option to disallow 'namenode format -force'
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193948#comment-16193948
]
Ajay Kumar commented on HDFS-12420:
---
[~arpitagarwal] thanks for review and commit.
[~shahrs87],[~daryn],[~aw],[~vagarychen],[~anu],[~vinayrpet],[~kihwal] thanks
for review and valuable feedback.
> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch,
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch,
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch,
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch,
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of
> Namenode in production cluster. If someone really wants to delete the
> complete fsImage, they can first delete the metadata dir and then run {code}
> hdfs namenode -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Add an option to disallow 'namenode format -force'
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193945#comment-16193945
]
Ajay Kumar commented on HDFS-12420:
---
[~arpitagarwal] thanks for review and commit.
[~shahrs87],[~daryn],[~aw],[~vagarychen],[~anu],[~vinayrpet],[~kihwal] thanks
for review and valuable feedback.
> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch,
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch,
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch,
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch,
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of
> Namenode in production cluster. If someone really wants to delete the
> complete fsImage, they can first delete the metadata dir and then run {code}
> hdfs namenode -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12567) BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes
[
https://issues.apache.org/jira/browse/HDFS-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-12567:
---
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.0, thanks for the review Eddy!
> BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes
> --
>
> Key: HDFS-12567
> URL: https://issues.apache.org/jira/browse/HDFS-12567
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0
>
> Attachments: HDFS-12567.001.patch, HDFS-12567.002.patch,
> HDFS-12567.003.patch, HDFS-12567.repro.patch
>
>
> Found this while doing some testing on an internal cluster with an unusual
> setup. We have a rack with ~20 nodes, then a few more with just a few nodes.
> It would fail to get (# data blocks) datanodes even though there were plenty
> of DNs on the rack with 20 DNs.
> I managed to reproduce this same issue in a unit test, stack trace like this:
> {noformat}
> java.io.IOException: File /testfile0 could only be written to 5 of the 6
> required nodes for RS-6-3-1024k. There are 9 datanode(s) running and no
> node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2083)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:286)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2609)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:863)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:548)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> {noformat}
> This isn't a very critical bug since it's an unusual rack configuration, but
> it can easily happen during testing.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12273) Federation UI
[ https://issues.apache.org/jira/browse/HDFS-12273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193925#comment-16193925 ] Chris Douglas commented on HDFS-12273: -- Ran the test-patch profile and skimmed the output. I don't understand where those warnings are coming from, either. A hard-coded 1s timeout seems a tad unforgiving, but if that's reasonable in practice then OK. +1 > Federation UI > - > > Key: HDFS-12273 > URL: https://issues.apache.org/jira/browse/HDFS-12273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri > Fix For: HDFS-10467 > > Attachments: federationUI-1.png, federationUI-2.png, > federationUI-3.png, HDFS-12273-HDFS-10467-000.patch, > HDFS-12273-HDFS-10467-001.patch, HDFS-12273-HDFS-10467-002.patch, > HDFS-12273-HDFS-10467-003.patch, HDFS-12273-HDFS-10467-004.patch, > HDFS-12273-HDFS-10467-005.patch, HDFS-12273-HDFS-10467-006.patch, > HDFS-12273-HDFS-10467-007.patch, HDFS-12273-HDFS-10467-008.patch, > HDFS-12273-HDFS-10467-009.patch, HDFS-12273-HDFS-10467-010.patch > > > Add the Web UI to the Router to expose the status of the federated cluster. > It includes the federation metrics. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193915#comment-16193915 ] Bharat Viswanadham edited comment on HDFS-12582 at 10/5/17 11:40 PM: - [~vagarychen] Updated the patch v02. Fixed UT failures and checkstyle issues. Test failures is due to error in code, fixed it. Tested locally. Test cases are passing. was (Author: bharatviswa): [~vagarychen] Updated the patch v02. Test failures is due to error in code. Tested locally. Test cases are passing. > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12582: -- Attachment: HDFS-12582.02.patch > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12599) Move DataNodeTestUtils.mockDatanodeBlkPinning into mock test util class
[
https://issues.apache.org/jira/browse/HDFS-12599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193917#comment-16193917
]
Hadoop QA commented on HDFS-12599:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 71 unchanged - 0 fixed = 74 total (was 71) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 39s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020 |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890583/HDFS-12599.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b1c8ee4cd304 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Updated] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12582: -- Attachment: (was: HDFS-12582.02.patch) > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193915#comment-16193915 ] Bharat Viswanadham commented on HDFS-12582: --- [~vagarychen] Updated the patch v02. Test failures is due to error in code. Tested locally. Test cases are passing. > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12582) Replace HdfsFileStatus constructor with a builder pattern.
[ https://issues.apache.org/jira/browse/HDFS-12582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-12582: -- Attachment: HDFS-12582.02.patch > Replace HdfsFileStatus constructor with a builder pattern. > -- > > Key: HDFS-12582 > URL: https://issues.apache.org/jira/browse/HDFS-12582 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham > Attachments: HDFS-12582.01.patch, HDFS-12582.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10659) Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory
[
https://issues.apache.org/jira/browse/HDFS-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193895#comment-16193895
]
Hadoop QA commented on HDFS-10659:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 6 new + 238 unchanged - 15 fixed = 244 total (was 253) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 49s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
50s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}112m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.qjournal.server.Journal.jnSyncer; locked 80% of time
Unsynchronized access at Journal.java:80% of time Unsynchronized access at
Journal.java:[line 1155] |
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-10659 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890584/HDFS-10659.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 8cbd7f9d98f4 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d0046bb |
[jira] [Commented] (HDFS-12502) nntop should support a category based on FilesInGetListingOps
[
https://issues.apache.org/jira/browse/HDFS-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193893#comment-16193893
]
Hadoop QA commented on HDFS-12502:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 45s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12502 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890586/HDFS-12502.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux ed4124ce34ea 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d0046bb |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/21544/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21544/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21544/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> nntop
[jira] [Updated] (HDFS-12411) Ozone: Add container usage information to DN container report
[ https://issues.apache.org/jira/browse/HDFS-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-12411: -- Attachment: HDFS-12411-HDFS-7240.005.patch Thanks [~vagarychen], update a rebased patch. > Ozone: Add container usage information to DN container report > - > > Key: HDFS-12411 > URL: https://issues.apache.org/jira/browse/HDFS-12411 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, scm >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Labels: ozoneMerge > Attachments: HDFS-12411-HDFS-7240.001.patch, > HDFS-12411-HDFS-7240.002.patch, HDFS-12411-HDFS-7240.003.patch, > HDFS-12411-HDFS-7240.004.patch, HDFS-12411-HDFS-7240.005.patch > > > Current DN ReportState for container only has a counter, we will need to > include individual container usage information so that SCM can > * close container when they are full > * assign container for block service with different policies. > * etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12411) Ozone: Add container usage information to DN container report
[ https://issues.apache.org/jira/browse/HDFS-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193864#comment-16193864 ] Chen Liang commented on HDFS-12411: --- [~xyao] seems v004 patch needs to be rebased now. > Ozone: Add container usage information to DN container report > - > > Key: HDFS-12411 > URL: https://issues.apache.org/jira/browse/HDFS-12411 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, scm >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Labels: ozoneMerge > Attachments: HDFS-12411-HDFS-7240.001.patch, > HDFS-12411-HDFS-7240.002.patch, HDFS-12411-HDFS-7240.003.patch, > HDFS-12411-HDFS-7240.004.patch > > > Current DN ReportState for container only has a counter, we will need to > include individual container usage information so that SCM can > * close container when they are full > * assign container for block service with different policies. > * etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12420) Add an option to disallow 'namenode format -force'
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193861#comment-16193861
]
Hudson commented on HDFS-12420:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13036 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13036/])
HDFS-12420. Add an option to disallow 'namenode format -force'. (arp: rev
b6942cbe9b8c9469e8c2b64c3268d671f5a43e75)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeFormatException.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch,
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch,
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch,
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch,
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of
> Namenode in production cluster. If someone really wants to delete the
> complete fsImage, they can first delete the metadata dir and then run {code}
> hdfs namenode -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12604) StreamCapability enums are not displayed in javadoc
[
https://issues.apache.org/jira/browse/HDFS-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193859#comment-16193859
]
John Zhuge commented on HDFS-12604:
---
[~eddyxu] I recently attempted to make some changes in this area. Please take a
look at
https://issues.apache.org/jira/secure/attachment/12890478/HADOOP-14872.009.patch.
> StreamCapability enums are not displayed in javadoc
> ---
>
> Key: HDFS-12604
> URL: https://issues.apache.org/jira/browse/HDFS-12604
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
>
> http://hadoop.apache.org/docs/r3.0.0-beta1/api/org/apache/hadoop/fs/StreamCapabilities.html
> {{StreamCapability#HFLUSH}} and {{StreamCapability#HSYNC}} are not displayed
> in the doc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12420) Add an option to disallow 'namenode format -force'
[
https://issues.apache.org/jira/browse/HDFS-12420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-12420:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to trunk after verifying the unit test failures are unrelated.
Thanks for the contribution [~ajayydv], and thanks everyone else for the ideas
and code reviews.
> Add an option to disallow 'namenode format -force'
> --
>
> Key: HDFS-12420
> URL: https://issues.apache.org/jira/browse/HDFS-12420
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Attachments: HDFS-12420.01.patch, HDFS-12420.02.patch,
> HDFS-12420.03.patch, HDFS-12420.04.patch, HDFS-12420.05.patch,
> HDFS-12420.06.patch, HDFS-12420.07.patch, HDFS-12420.08.patch,
> HDFS-12420.09.patch, HDFS-12420.10.patch, HDFS-12420.11.patch,
> HDFS-12420.12.patch
>
>
> Support for disabling NameNode format to avoid accidental formatting of
> Namenode in production cluster. If someone really wants to delete the
> complete fsImage, they can first delete the metadata dir and then run {code}
> hdfs namenode -format{code} manually.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12567) BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes
[
https://issues.apache.org/jira/browse/HDFS-12567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193841#comment-16193841
]
Hadoop QA commented on HDFS-12567:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 48s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 95m
29s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | HDFS-12567 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12890572/HDFS-12567.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c19be1be40c1 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d0046bb |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/21538/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/21538/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes
> --
>
> Key: HDFS-12567
> URL: https://issues.apache.org/jira/browse/HDFS-12567
> Project: Hadoop HDFS
>
[jira] [Commented] (HDFS-11146) Excess replicas will not be deleted until all storages's FBR received after failover
[
https://issues.apache.org/jira/browse/HDFS-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193833#comment-16193833
]
Daryn Sharp commented on HDFS-11146:
Thanks for the update but it's still in the common code path which adds latency
for every single heartbeat. I think checking in
{{HeartbeatManager#heartbeatCheck}} is more appropriate. It runs less often,
and it already iterates the storages checking for staleness too.
> Excess replicas will not be deleted until all storages's FBR received after
> failover
>
>
> Key: HDFS-11146
> URL: https://issues.apache.org/jira/browse/HDFS-11146
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Brahma Reddy Battula
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-11146-002.patch, HDFS-11146-003.patch,
> HDFS-11146-004.patch, HDFS-11146.patch
>
>
> Excess replicas will not be deleted until all storages's FBR received after
> failover.
> Thinking following soultion can help.
> *Solution:*
> I think after failover, As DNs aware of failover ,so they can send another
> block report (FBR) irrespective of interval.May be some shuffle can be done,
> similar to initial delay.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12604) StreamCapability enums are not displayed in javadoc
Lei (Eddy) Xu created HDFS-12604:
Summary: StreamCapability enums are not displayed in javadoc
Key: HDFS-12604
URL: https://issues.apache.org/jira/browse/HDFS-12604
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs
Affects Versions: 3.0.0-beta1
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Priority: Minor
http://hadoop.apache.org/docs/r3.0.0-beta1/api/org/apache/hadoop/fs/StreamCapabilities.html
{{StreamCapability#HFLUSH}} and {{StreamCapability#HSYNC}} are not displayed in
the doc.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12494) libhdfs SIGSEGV in setTLSExceptionStrings
[
https://issues.apache.org/jira/browse/HDFS-12494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193827#comment-16193827
]
Sailesh Mukil commented on HDFS-12494:
--
[~jzhuge] Thanks for submitting the patch.
One question I have:
Shouldn't we now clear the TLS if a failure happens in getGlobalJNIEnv(), since
we now set the TLS before calling getGlobalJNIEnv(), but that TLS has no
associated JNIEnv? So the next time that thread calls getJNIEnv, it would get a
TLS without a JNIEnv which would fail elsewhere.
> libhdfs SIGSEGV in setTLSExceptionStrings
> -
>
> Key: HDFS-12494
> URL: https://issues.apache.org/jira/browse/HDFS-12494
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.0.0-alpha4
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HDFS-12494.001.patch
>
>
> libhdfs application crashes when CLASSPATH is set but not set properly. It
> uses wildcard in this case.
> {noformat}
> $ export CLASSPATH=$(hadoop classpath)
> $ pwd
> /Users/jzhuge/hadoop2/hadoop-hdfs-project/hadoop-hdfs/target/native
> $ ./test_libhdfs_ops
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x0001052968f7, pid=14147, tid=775
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_79-b15) (build
> 1.7.0_79-b15)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode bsd-amd64
> compressed oops)
> # Problematic frame:
> # C [libhdfs.0.0.0.dylib+0x38f7] setTLSExceptionStrings+0x47
> #
> # Core dump written. Default location: /cores/core or core.14147
> #
> # An error report file with more information is saved as:
> #
> /Users/jzhuge/hadoop2/hadoop-hdfs-project/hadoop-hdfs/target/native/hs_err_pid14147.log
> #
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> Abort trap: 6 (core dumped)
> [jzhuge@jzhuge-MBP native]((be32925fff5...) *+)$ lldb -c /cores/core.14147
> (lldb) target create --core "/cores/core.14147"
> warning: (x86_64) /cores/core.14147 load command 549 LC_SEGMENT_64 has a
> fileoff + filesize (0x14627f000) that extends beyond the end of the file
> (0x14627e000), the segment will be truncated to match
> warning: (x86_64) /cores/core.14147 load command 550 LC_SEGMENT_64 has a
> fileoff (0x14627f000) that extends beyond the end of the file (0x14627e000),
> ignoring this section
> Core file '/cores/core.14147' (x86_64) was loaded.
> (lldb) bt
> * thread #1, stop reason = signal SIGSTOP
> * frame #0: 0x7fffcf89ad42 libsystem_kernel.dylib`__pthread_kill + 10
> frame #1: 0x7fffcf988457 libsystem_pthread.dylib`pthread_kill + 90
> frame #2: 0x7fffcf800420 libsystem_c.dylib`abort + 129
> frame #3: 0x0001056cd5fb libjvm.dylib`os::abort(bool) + 25
> frame #4: 0x0001057d98fc libjvm.dylib`VMError::report_and_die() + 2308
> frame #5: 0x0001056cefb5 libjvm.dylib`JVM_handle_bsd_signal + 1083
> frame #6: 0x7fffcf97bb3a libsystem_platform.dylib`_sigtramp + 26
> frame #7: 0x0001052968f8
> libhdfs.0.0.0.dylib`setTLSExceptionStrings(rootCause=0x,
> stackTrace=0x) at jni_helper.c:589 [opt]
> frame #8: 0x0001052954f0
> libhdfs.0.0.0.dylib`printExceptionAndFreeV(env=0x7ffaff0019e8,
> exc=0x7ffafec04140, noPrintFlags=, fmt="loadFileSystems",
> ap=) at exception.c:183 [opt]
> frame #9: 0x0001052956bb
> libhdfs.0.0.0.dylib`printExceptionAndFree(env=,
> exc=, noPrintFlags=, fmt=) at
> exception.c:213 [opt]
> frame #10: 0x0001052967f4 libhdfs.0.0.0.dylib`getJNIEnv [inlined]
> getGlobalJNIEnv at jni_helper.c:463 [opt]
> frame #11: 0x00010529664f libhdfs.0.0.0.dylib`getJNIEnv at
> jni_helper.c:528 [opt]
> frame #12: 0x0001052975eb
> libhdfs.0.0.0.dylib`hdfsBuilderConnect(bld=0x7ffafed0) at hdfs.c:693
> [opt]
> frame #13: 0x00010528be30 test_libhdfs_ops`main(argc=,
> argv=) at test_libhdfs_ops.c:91 [opt]
> frame #14: 0x7fffcf76c235 libdyld.dylib`start + 1
> (lldb) f 10
> libhdfs.0.0.0.dylib was compiled with optimization - stepping may behave
> oddly; variables may not be available.
> frame #10: 0x0001052967f4 libhdfs.0.0.0.dylib`getJNIEnv [inlined]
> getGlobalJNIEnv at jni_helper.c:463 [opt]
>460 "org/apache/hadoop/fs/FileSystem",
>461 "loadFileSystems", "()V");
>462if (jthr) {
> -> 463printExceptionAndFree(env, jthr, PRINT_EXC_ALL,
> "loadFileSystems");
>464}
>465}
>466else {
>
[jira] [Commented] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193820#comment-16193820 ] Xiao Chen commented on HDFS-12603: -- +1 based on comments from HDFS-7964. > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Status: Patch Available (was: Open) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0-alpha1, 2.8.0, 2.9.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Target Version/s: 2.9.0, 2.8.3, 3.0.0 (was: 2.9.0, 3.0.0) > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12603) Enable async edit logging by default
[ https://issues.apache.org/jira/browse/HDFS-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12603: --- Attachment: HDFS-12603.001.patch Trivial patch attached. The original patches in HDFS-7964 parameterized all the edit log related tests to run with async edit logging enabled and disabled, so I don't think we need any additional test coverage. > Enable async edit logging by default > > > Key: HDFS-12603 > URL: https://issues.apache.org/jira/browse/HDFS-12603 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.8.0, 2.9.0, 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-12603.001.patch > > > HDFS-7964 added support for async edit logging. Based on further discussion > in that JIRA, we think it's safe to turn this on by default for better > out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193812#comment-16193812 ] Xiaoyu Yao commented on HDFS-12600: --- Agree, thanks [~xiaochen] for the new patch to trigger the hadoop-hdfs test. > CLONE - KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HDFS-12600 > URL: https://issues.apache.org/jira/browse/HDFS-12600 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-12600.001.patch, HDFS-12600.002.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12543) Ozone : allow create key without specifying size
[ https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-12543: -- Attachment: HDFS-12543-HDFS-7240.012.patch Update v012 patch to fix checkstyle warning. > Ozone : allow create key without specifying size > > > Key: HDFS-12543 > URL: https://issues.apache.org/jira/browse/HDFS-12543 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Labels: ozoneMerge > Attachments: HDFS-12543-HDFS-7240.001.patch, > HDFS-12543-HDFS-7240.002.patch, HDFS-12543-HDFS-7240.003.patch, > HDFS-12543-HDFS-7240.004.patch, HDFS-12543-HDFS-7240.005.patch, > HDFS-12543-HDFS-7240.006.patch, HDFS-12543-HDFS-7240.007.patch, > HDFS-12543-HDFS-7240.008.patch, HDFS-12543-HDFS-7240.009.patch, > HDFS-12543-HDFS-7240.010.patch, HDFS-12543-HDFS-7240.011.patch, > HDFS-12543-HDFS-7240.012.patch > > > Currently when creating a key, it is required to specify the total size of > the key. This makes it inconvenient for the case where a key is created and > data keeps coming and being appended. This JIRA is remove the requirement of > specifying the size on key creation, and allows appending to the key > indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HDFS-12600: - Attachment: HDFS-12600.002.patch > CLONE - KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HDFS-12600 > URL: https://issues.apache.org/jira/browse/HDFS-12600 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-12600.001.patch, HDFS-12600.002.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12598) Ozone: Fix 3 node ratis replication in Ozone
[ https://issues.apache.org/jira/browse/HDFS-12598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193807#comment-16193807 ] Anu Engineer commented on HDFS-12598: - [~msingh] Thanks for getting this to work. Ozone changes look good to me. I will wait for [~szetszwo] to comment on Ratis changes. > Ozone: Fix 3 node ratis replication in Ozone > > > Key: HDFS-12598 > URL: https://issues.apache.org/jira/browse/HDFS-12598 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Labels: ozoneMerge > Fix For: HDFS-7240 > > Attachments: HDFS-12598-HDFS-7240.001.patch > > > Enabling ratis 3 node replication on ozone currently fails because > XceiverServerRatis doesn't initialize correctly because the server id is not > initialized correctly. Uploading an initial patch for early feedback. Will > upload an updated patch after testing on the cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7964) Add support for async edit logging
[ https://issues.apache.org/jira/browse/HDFS-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193806#comment-16193806 ] Andrew Wang commented on HDFS-7964: --- Thanks Kihwal, filed HDFS-12603 to make this happen. > Add support for async edit logging > -- > > Key: HDFS-7964 > URL: https://issues.apache.org/jira/browse/HDFS-7964 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.2-alpha >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.8.0, 2.9.0, 3.0.0-alpha1 > > Attachments: HDFS-7964-branch-2.7.patch, > HDFS-7964-branch-2.8.0.patch, HDFS-7964.patch, HDFS-7964.patch, > HDFS-7964.patch, HDFS-7964.patch, HDFS-7964-rebase.patch > > > Edit logging is a major source of contention within the NN. LogEdit is > called within the namespace write log, while logSync is called outside of the > lock to allow greater concurrency. The handler thread remains busy until > logSync returns to provide the client with a durability guarantee for the > response. > Write heavy RPC load and/or slow IO causes handlers to stall in logSync. > Although the write lock is not held, readers are limited/starved and the call > queue fills. Combining an edit log thread with postponed RPC responses from > HADOOP-10300 will provide the same durability guarantee but immediately free > up the handlers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12600) CLONE - KMSClientProvider won't work with KMS delegation token retrieved from non-Java client.
[ https://issues.apache.org/jira/browse/HDFS-12600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193808#comment-16193808 ] Xiao Chen commented on HDFS-12600: -- +1 unit7m 54s hadoop-common in the patch passed. Hi [~xyao], thanks for the jira. I think we still need the hdfs dummy change to let jenkins actually run that component. Attaching a patch for that... > CLONE - KMSClientProvider won't work with KMS delegation token retrieved from > non-Java client. > -- > > Key: HDFS-12600 > URL: https://issues.apache.org/jira/browse/HDFS-12600 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-12600.001.patch > > > HADOOP-13381 added support to use KMS delegation token to connect to KMS > server for key operations. However, the logic to check if the UGI container > KMS delegation token assumes that the token must contain a service attribute. > Otherwise, a KMS delegation token won't be recognized. > For delegation token obtained via non-java client such curl (http), the > default DelegationTokenAuthenticationHandler only support *renewer* parameter > and assume the client itself will add the service attribute. This makes a > java client with KMSClientProvdier can't use for KMS delegation token > retrieved form non-java client because the token does not contain a service > attribute. > I did some investigation on this and found two solutions: > 1. Similar use case exists for webhdfs, and webhdfs supports it with a > ["service" > parameter|https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Get_Delegation_Token]. > We can do this similarly by allowing client to specify a service attribute in > the request URL and included in the token returned like webhdfs. Even though > this will change in DelegationTokenAuthenticationHandler and may affect many > other web component, this seems to be a clean and low risk solution because > it will be an optional parameter. Also, other components get non-java client > interop support for free if they have the similar use case. > 2. The other way to solve this is to release the token check in > KMSClientProvider to check only the token kind instead of the service. This > is an easy work around but seems less optimal to me. > cc: [~xiaochen] for additional input. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12603) Enable async edit logging by default
Andrew Wang created HDFS-12603: -- Summary: Enable async edit logging by default Key: HDFS-12603 URL: https://issues.apache.org/jira/browse/HDFS-12603 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0-alpha1, 2.8.0, 2.9.0 Reporter: Andrew Wang Assignee: Andrew Wang HDFS-7964 added support for async edit logging. Based on further discussion in that JIRA, we think it's safe to turn this on by default for better out-of-the-box performance. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12543) Ozone : allow create key without specifying size
[
https://issues.apache.org/jira/browse/HDFS-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16193802#comment-16193802
]
Hadoop QA commented on HDFS-12543:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new +
2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 41s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
26s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.ozone.scm.TestAllocateContainer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
