[jira] [Commented] (HDFS-10764) Fix INodeFile#getBlocks to not return null
[
https://issues.apache.org/jira/browse/HDFS-10764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429238#comment-15429238
]
Hudson commented on HDFS-10764:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10315 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10315/])
HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by (jing9:
rev 0faee62a0c8c1b8fd83227babfd00fbc2b26bddf)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
> Fix INodeFile#getBlocks to not return null
> --
>
> Key: HDFS-10764
> URL: https://issues.apache.org/jira/browse/HDFS-10764
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-10764.01.patch
>
>
> Not all callers of INodeFile#getBlocks check for null. e.g.
> {code}
> public final QuotaCounts storagespaceConsumedContiguous(
> BlockStoragePolicy bsp) {
> ...
> // Collect all distinct blocks
> Set allBlocks = new HashSet<>(Arrays.asList(getBlocks()));
> {code}
> We can either fix each caller or alternatively fix getBlocks to never return
> null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10764) Fix INodeFile#getBlocks to not return null
[
https://issues.apache.org/jira/browse/HDFS-10764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-10764:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Target Version/s: (was: 3.0.0-alpha1)
Status: Resolved (was: Patch Available)
The failed tests should be unrelated. I've committed the patch to trunk,
branch-2, and branch-2.8.
> Fix INodeFile#getBlocks to not return null
> --
>
> Key: HDFS-10764
> URL: https://issues.apache.org/jira/browse/HDFS-10764
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-10764.01.patch
>
>
> Not all callers of INodeFile#getBlocks check for null. e.g.
> {code}
> public final QuotaCounts storagespaceConsumedContiguous(
> BlockStoragePolicy bsp) {
> ...
> // Collect all distinct blocks
> Set allBlocks = new HashSet<>(Arrays.asList(getBlocks()));
> {code}
> We can either fix each caller or alternatively fix getBlocks to never return
> null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10764) Fix INodeFile#getBlocks to not return null
[
https://issues.apache.org/jira/browse/HDFS-10764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429234#comment-15429234
]
Jing Zhao commented on HDFS-10764:
--
And thanks a lot for the contribution, [~arpitagarwal]!
> Fix INodeFile#getBlocks to not return null
> --
>
> Key: HDFS-10764
> URL: https://issues.apache.org/jira/browse/HDFS-10764
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.0.0-alpha2
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.8.0
>
> Attachments: HDFS-10764.01.patch
>
>
> Not all callers of INodeFile#getBlocks check for null. e.g.
> {code}
> public final QuotaCounts storagespaceConsumedContiguous(
> BlockStoragePolicy bsp) {
> ...
> // Collect all distinct blocks
> Set allBlocks = new HashSet<>(Arrays.asList(getBlocks()));
> {code}
> We can either fix each caller or alternatively fix getBlocks to never return
> null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9392) Admins support for maintenance state
[
https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429200#comment-15429200
]
Hadoop QA commented on HDFS-9392:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
399 unchanged - 15 fixed = 402 total (was 414) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 51s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824681/HDFS-9392-2.patch |
| JIRA Issue | HDFS-9392 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 52817a9c79ff 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2da32a6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16487/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16487/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Updated] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-8986:
Attachment: HDFS-8986.05.patch
Thanks Wei-Chiu, patch 5 attached to rename the params.
> Add option to -du to calculate directory space usage excluding snapshots
>
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch,
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its
> children), the report includes space consumed by blocks that are only present
> in the snapshots. This is confusing for end users.
> {noformat}
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0 0 /tmp/parent
> 0 0 /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related
> disk usage in the output
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429191#comment-15429191
]
Wei-Chiu Chuang commented on HDFS-8986:
---
Hello [~xiaochen], thanks again for the new patch. It looks mostly good to me.
One minor issue:
The parameter of {{ContentSummary#Builder}} methods needs meaningful names. So
instead of
{code}
public Builder snapshotLength(long val) {
this.snapshotLength = val;
{code}
The parameter val should be named {{snapshotLength}} or simply {{length}}.
> Add option to -du to calculate directory space usage excluding snapshots
>
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch,
> HDFS-8986.03.patch, HDFS-8986.04.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its
> children), the report includes space consumed by blocks that are only present
> in the snapshots. This is confusing for end users.
> {noformat}
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0 0 /tmp/parent
> 0 0 /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related
> disk usage in the output
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10692) Point JDiff base version for HDFS from 2.6.0 to 2.7.2
[
https://issues.apache.org/jira/browse/HDFS-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429184#comment-15429184
]
Hadoop QA commented on HDFS-10692:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 201 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824492/HDFS-10692.2.patch |
| JIRA Issue | HDFS-10692 |
| Optional Tests | asflicense xml |
| uname | Linux d35ca0b2934f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 99603e9 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16488/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16488/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Point JDiff base version for HDFS from 2.6.0 to 2.7.2
> -
>
> Key: HDFS-10692
> URL: https://issues.apache.org/jira/browse/HDFS-10692
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: 3.0.0-alpha1-jdiff-hdfs.zip, HDFS-10692.1.patch,
> HDFS-10692.2.patch
>
>
> Now JDiff is pointed to 2.6.0, we need to upgrade it to the latest stable
> release (2.7.2)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10692) Point JDiff base version for HDFS from 2.6.0 to 2.7.2
[ https://issues.apache.org/jira/browse/HDFS-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-10692: --- Assignee: Wangda Tan Hadoop Flags: Reviewed Status: Patch Available (was: Open) Straight forward patch with generated jdiff. Will check this in if Jenkins says okay.. > Point JDiff base version for HDFS from 2.6.0 to 2.7.2 > - > > Key: HDFS-10692 > URL: https://issues.apache.org/jira/browse/HDFS-10692 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: 3.0.0-alpha1-jdiff-hdfs.zip, HDFS-10692.1.patch, > HDFS-10692.2.patch > > > Now JDiff is pointed to 2.6.0, we need to upgrade it to the latest stable > release (2.7.2) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9392) Admins support for maintenance state
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Attachment: HDFS-9392-2.patch Updated patch to fix checkstyle, whitespace and unit tests. > Admins support for maintenance state > > > Key: HDFS-9392 > URL: https://issues.apache.org/jira/browse/HDFS-9392 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ming Ma > Attachments: HDFS-9392-2.patch, HDFS-9392.patch > > > This is to allow admins to put nodes into maintenance state with optional > timeout value as well as take nodes out of maintenance state. Likely we will > leverage what we come up in https://issues.apache.org/jira/browse/HDFS-9005. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429100#comment-15429100
]
Hadoop QA commented on HDFS-10702:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 40s{color} | {color:orange} root: The patch generated 10 new + 1027
unchanged - 2 fixed = 1037 total (was 1029) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
57s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs generated 10 new + 7
unchanged - 0 fixed = 17 total (was 7) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
23s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824658/HDFS-10702.005.patch |
| JIRA Issue | HDFS-10702 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux acbac0ae1cd5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 723facf |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16486/artifact/patchprocess/diff-checkstyle-root.txt
|
| javadoc |
[jira] [Comment Edited] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429061#comment-15429061
]
Lei (Eddy) Xu edited comment on HDFS-10636 at 8/19/16 11:46 PM:
bq. If not, are there existing cases where vendors might be using
FinalizedReplica for non-File backed data?
Yea. I know vendors who have already implemented similar solutions based on
{{FinalizedReplica}}, because {{ProvidedReplica}} did not exist back to the
days. But I think it might be OK for them if this patch did not break their
code.
Another option might be that, "Finalized" looks like a state in the pipeline
state machine, instead of a "Location" where the replica is stored, IMO. Have
you considered about making {{LocalReplica}} and {{ProvidedReplica}} being
subclasses of {{FinalizedReplica}}? So that for the pipeline state machine part
of code, all the existing code can re-use the same {{FinalizedReplica}} as what
it is today. Would love to hear your thoughts about it.
bq. Having a good test coverage seems a cleaner way to solve this.
Agree. That would be a nice thing to do.
was (Author: eddyxu):
bq. If not, are there existing cases where vendors might be using
FinalizedReplica for non-File backed data?
Yea. I know vendors who have already implemented similar solutions based on
{{FinalizedReplica}}, because {{ProvidedReplica}} did not exist back to the
days. But I think it might be OK for them if this patch did not break the
their code.
Another option might be that, "Finalized" looks like a state in the pipeline
state machine, instead of a "Location" where the replica is stored, IMO. Have
you considered about making {{LocalReplica}} and {{ProvidedReplica}} being
subclasses of {{FinalizedReplica}}? So that for the pipeline state machine part
of code, all the existing code can re-use the same {{FinalizedReplica}} as what
it is today. Would love to hear your thoughts about it.
bq. Having a good test coverage seems a cleaner way to solve this.
Agree. That would be a nice thing to do.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429061#comment-15429061
]
Lei (Eddy) Xu commented on HDFS-10636:
--
bq. If not, are there existing cases where vendors might be using
FinalizedReplica for non-File backed data?
Yea. I know vendors who have already implemented similar solutions based on
{{FinalizedReplica}}, because {{ProvidedReplica}} did not exist back to the
days. But I think it might be OK for them if this patch did not break the
their code.
Another option might be that, "Finalized" looks like a state in the pipeline
state machine, instead of a "Location" where the replica is stored, IMO. Have
you considered about making {{LocalReplica}} and {{ProvidedReplica}} being
subclasses of {{FinalizedReplica}}? So that for the pipeline state machine part
of code, all the existing code can re-use the same {{FinalizedReplica}} as what
it is today. Would love to hear your thoughts about it.
bq. Having a good test coverage seems a cleaner way to solve this.
Agree. That would be a nice thing to do.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10608) Include event for AddBlock in Inotify Event Stream
[
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429063#comment-15429063
]
Hadoop QA commented on HDFS-10608:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project: The patch generated 11 new
+ 142 unchanged - 1 fixed = 153 total (was 143) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824649/HDFS-10608.v2.patch |
| JIRA Issue | HDFS-10608 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux d9a2fd09c10e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 763f049 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16485/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429014#comment-15429014 ] Jiayi Zhou commented on HDFS-10702: --- Test failed due to an improper default value for StaleBound. Upload a new patch to fill the problem. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch, > StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.005.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, HDFS-10702.005.patch, > StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[
https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428971#comment-15428971
]
Hadoop QA commented on HDFS-10702:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 39s{color} | {color:orange} root: The patch generated 10 new + 1027
unchanged - 2 fixed = 1037 total (was 1029) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
12s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 13s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestQuotasWithHA |
| | hadoop.hdfs.server.namenode.ha.TestXAttrsWithHA |
| | hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.server.namenode.ha.TestStandbyIsHot |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824629/HDFS-10702.004.patch |
| JIRA Issue | HDFS-10702 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 5510bd6b69c2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 763f049 |
| Default
[jira] [Updated] (HDFS-10608) Include event for AddBlock in Inotify Event Stream
[ https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HDFS-10608: -- Status: Patch Available (was: Open) > Include event for AddBlock in Inotify Event Stream > -- > > Key: HDFS-10608 > URL: https://issues.apache.org/jira/browse/HDFS-10608 > Project: Hadoop HDFS > Issue Type: Task >Reporter: churro morales >Priority: Minor > Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, > HDFS-10608.v2.patch > > > It would be nice to have an AddBlockEvent in the INotify pipeline. Based on > discussions from mailing list: > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10608) Include event for AddBlock in Inotify Event Stream
[ https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HDFS-10608: -- Attachment: HDFS-10608.v2.patch Removed the blockPoolId field and no more exposing Blocks from the AddBlockEvent API. > Include event for AddBlock in Inotify Event Stream > -- > > Key: HDFS-10608 > URL: https://issues.apache.org/jira/browse/HDFS-10608 > Project: Hadoop HDFS > Issue Type: Task >Reporter: churro morales >Priority: Minor > Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, > HDFS-10608.v2.patch > > > It would be nice to have an AddBlockEvent in the INotify pipeline. Based on > discussions from mailing list: > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10608) Include event for AddBlock in Inotify Event Stream
[ https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HDFS-10608: -- Status: Open (was: Patch Available) > Include event for AddBlock in Inotify Event Stream > -- > > Key: HDFS-10608 > URL: https://issues.apache.org/jira/browse/HDFS-10608 > Project: Hadoop HDFS > Issue Type: Task >Reporter: churro morales >Priority: Minor > Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, > HDFS-10608.v2.patch > > > It would be nice to have an AddBlockEvent in the INotify pipeline. Based on > discussions from mailing list: > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7959) WebHdfs logging is missing on Datanode
[ https://issues.apache.org/jira/browse/HDFS-7959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428774#comment-15428774 ] Rushabh S Shah commented on HDFS-7959: -- Thank you !! > WebHdfs logging is missing on Datanode > -- > > Key: HDFS-7959 > URL: https://issues.apache.org/jira/browse/HDFS-7959 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS-7959.1.branch-2.patch, HDFS-7959.1.trunk.patch, > HDFS-7959.2.branch-2.patch, HDFS-7959.2.trunk.patch, > HDFS-7959.3.branch-2.patch, HDFS-7959.3.trunk.patch, > HDFS-7959.branch-2.patch, HDFS-7959.patch, HDFS-7959.patch, HDFS-7959.patch, > HDFS-7959.trunk.patch > > > After the conversion to netty, webhdfs requests are not logged on datanodes. > The existing jetty log only logs the non-webhdfs requests that come through > the internal proxy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428772#comment-15428772 ] Jiayi Zhou edited comment on HDFS-10702 at 8/19/16 8:35 PM: Upload a patch to fix some of the issues commented by Andrew. I haven't worked on the RpcHeader modification yet, want to see if the new patch runs properly first. * Fix nitty issues. * Remove MethodCategory and use annotations instead. Add a new annotation ReadOnly. * Idea of SyncInfo is also to add more fields in the future. NamenodeProtocol#getTransactionID is commented as NameNodeProtocol rather that ClientProtocol, so I add getSyncInfo() for ClientProtocol. Remove state from SyncInfo because we no longer need it. was (Author: clouderajiayi): Upload a path to fix some of the issues commented by Andrew. I haven't worked on the RpcHeader modification yet, want to see if the new patch runs properly first. * Fix nitty issues. * Remove MethodCategory and use annotations instead. Add a new annotation ReadOnly. * Idea of SyncInfo is also to add more fields in the future. NamenodeProtocol#getTransactionID is commented as NameNodeProtocol rather that ClientProtocol, so I add getSyncInfo() for ClientProtocol. Remove state from SyncInfo because we no longer need it. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-7959) WebHdfs logging is missing on Datanode
[ https://issues.apache.org/jira/browse/HDFS-7959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-7959: - Fix Version/s: (was: 2.9.0) 2.8.0 > WebHdfs logging is missing on Datanode > -- > > Key: HDFS-7959 > URL: https://issues.apache.org/jira/browse/HDFS-7959 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS-7959.1.branch-2.patch, HDFS-7959.1.trunk.patch, > HDFS-7959.2.branch-2.patch, HDFS-7959.2.trunk.patch, > HDFS-7959.3.branch-2.patch, HDFS-7959.3.trunk.patch, > HDFS-7959.branch-2.patch, HDFS-7959.patch, HDFS-7959.patch, HDFS-7959.patch, > HDFS-7959.trunk.patch > > > After the conversion to netty, webhdfs requests are not logged on datanodes. > The existing jetty log only logs the non-webhdfs requests that come through > the internal proxy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7959) WebHdfs logging is missing on Datanode
[ https://issues.apache.org/jira/browse/HDFS-7959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428770#comment-15428770 ] Kihwal Lee commented on HDFS-7959: -- Cherry-picked to 2.8. > WebHdfs logging is missing on Datanode > -- > > Key: HDFS-7959 > URL: https://issues.apache.org/jira/browse/HDFS-7959 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.8.0 > > Attachments: HDFS-7959.1.branch-2.patch, HDFS-7959.1.trunk.patch, > HDFS-7959.2.branch-2.patch, HDFS-7959.2.trunk.patch, > HDFS-7959.3.branch-2.patch, HDFS-7959.3.trunk.patch, > HDFS-7959.branch-2.patch, HDFS-7959.patch, HDFS-7959.patch, HDFS-7959.patch, > HDFS-7959.trunk.patch > > > After the conversion to netty, webhdfs requests are not logged on datanodes. > The existing jetty log only logs the non-webhdfs requests that come through > the internal proxy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428772#comment-15428772 ] Jiayi Zhou commented on HDFS-10702: --- Upload a path to fix some of the issues commented by Andrew. I haven't worked on the RpcHeader modification yet, want to see if the new patch runs properly first. * Fix nitty issues. * Remove MethodCategory and use annotations instead. Add a new annotation ReadOnly. * Idea of SyncInfo is also to add more fields in the future. NamenodeProtocol#getTransactionID is commented as NameNodeProtocol rather that ClientProtocol, so I add getSyncInfo() for ClientProtocol. Remove state from SyncInfo because we no longer need it. > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10781) libhdfs++: redefine NN timeout to be "time without a response"
Bob Hansen created HDFS-10781: - Summary: libhdfs++: redefine NN timeout to be "time without a response" Key: HDFS-10781 URL: https://issues.apache.org/jira/browse/HDFS-10781 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Bob Hansen In the find tool, we submit a zillion requests to the NameNode asynchronously. As the queue on the NameNode grows, the time to response for each individual message will increase. In the find tool, we were eventually getting timeouts on requests, even though the NN was respoinding as fast as its little feet could carry it. I propose that we should redefine timeouts to be on a per-connection basis rather than per-request. If a client has an outstanding request to the NN but hasn't gotten a response back within n msec, it should declare the connection dead and retry. As long as the NameNode is being responsive to the best of its ability and providing data, we will not declare the link dead. One potential for Failure of Least Astonishment here is that it will mean any particular request from a client cannot be depended on to get a positive or negative response within a fixed amount of time, but I think that may be a good trade to make. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10702) Add a Client API and Proxy Provider to enable stale read from Standby
[ https://issues.apache.org/jira/browse/HDFS-10702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiayi Zhou updated HDFS-10702: -- Attachment: HDFS-10702.004.patch > Add a Client API and Proxy Provider to enable stale read from Standby > - > > Key: HDFS-10702 > URL: https://issues.apache.org/jira/browse/HDFS-10702 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jiayi Zhou >Assignee: Jiayi Zhou >Priority: Minor > Attachments: HDFS-10702.001.patch, HDFS-10702.002.patch, > HDFS-10702.003.patch, HDFS-10702.004.patch, StaleReadfromStandbyNN.pdf > > > Currently, clients must always talk to the active NameNode when performing > any metadata operation, which means active NameNode could be a bottleneck for > scalability. One way to solve this problem is to send read-only operations to > Standby NameNode. The disadvantage is that it might be a stale read. > Here, I'm thinking of adding a Client API to enable/disable stale read from > Standby which gives Client the power to set the staleness restriction. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-7959) WebHdfs logging is missing on Datanode
[ https://issues.apache.org/jira/browse/HDFS-7959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428722#comment-15428722 ] Rushabh S Shah commented on HDFS-7959: -- [~kihwal],[~sjlee0]: any reason why this change shouldn't go in branch-2.8 ? If it should then I can prepare a branch-2.8 patch. > WebHdfs logging is missing on Datanode > -- > > Key: HDFS-7959 > URL: https://issues.apache.org/jira/browse/HDFS-7959 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Labels: BB2015-05-TBR > Fix For: 2.9.0 > > Attachments: HDFS-7959.1.branch-2.patch, HDFS-7959.1.trunk.patch, > HDFS-7959.2.branch-2.patch, HDFS-7959.2.trunk.patch, > HDFS-7959.3.branch-2.patch, HDFS-7959.3.trunk.patch, > HDFS-7959.branch-2.patch, HDFS-7959.patch, HDFS-7959.patch, HDFS-7959.patch, > HDFS-7959.trunk.patch > > > After the conversion to netty, webhdfs requests are not logged on datanodes. > The existing jetty log only logs the non-webhdfs requests that come through > the internal proxy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9392) Admins support for maintenance state
[
https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428721#comment-15428721
]
Hadoop QA commented on HDFS-9392:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project: The patch generated 14 new
+ 402 unchanged - 12 fixed = 416 total (was 414) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 55s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.server.namenode.TestHostsFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824609/HDFS-9392.patch |
| JIRA Issue | HDFS-9392 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e984406d4205 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 03a9343 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16483/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| whitespace |
[jira] [Commented] (HDFS-10679) libhdfs++: Implement parallel find with wildcards tool
[
https://issues.apache.org/jira/browse/HDFS-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428659#comment-15428659
]
James Clampffer commented on HDFS-10679:
Thanks for the updates, new code looks good to me other than minor issue below.
One problem is that it looks like this reverses HDFS-10746. You might need to
just apply that patch to your branch if merging in HDFS-8707 isn't fixing it.
{code}
-uint64_t FileHandleImpl::get_bytes_read() { return bytes_read_.load(); }
+uint64_t FileHandleImpl::get_bytes_read() { return bytes_read_; }
-void FileHandleImpl::clear_bytes_read() { bytes_read_.store(0); }
+void FileHandleImpl::clear_bytes_read() { bytes_read_ = 0; }
{code}
> libhdfs++: Implement parallel find with wildcards tool
> --
>
> Key: HDFS-10679
> URL: https://issues.apache.org/jira/browse/HDFS-10679
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10679.HDFS-8707.000.patch,
> HDFS-10679.HDFS-8707.001.patch, HDFS-10679.HDFS-8707.002.patch,
> HDFS-10679.HDFS-8707.003.patch, HDFS-10679.HDFS-8707.004.patch,
> HDFS-10679.HDFS-8707.005.patch, HDFS-10679.HDFS-8707.006.patch,
> HDFS-10679.HDFS-8707.007.patch, HDFS-10679.HDFS-8707.008.patch,
> HDFS-10679.HDFS-8707.009.patch, HDFS-10679.HDFS-8707.010.patch,
> HDFS-10679.HDFS-8707.011.patch, HDFS-10679.HDFS-8707.012.patch
>
>
> The find tool will issue the GetListing namenode operation on a given
> directory, and filter the results using posix globbing library.
> If the recursive option is selected, for each returned entry that is a
> directory the tool will issue another asynchronous call GetListing and repeat
> the result processing in a recursive fashion.
> One implementation issue that needs to be addressed is the way how results
> are returned back to the user: we can either buffer the results and return
> them to the user in bulk, or we can return results continuously as they
> arrive. While buffering would be an easier solution, returning results as
> they arrive would be more beneficial to the user in terms of performance,
> since the result processing can start as soon as the first results arrive
> without any delay. In order to do that we need the user to use a loop to
> process arriving results, and we need to send a special message back to the
> user when the search is over.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10608) Include event for AddBlock in Inotify Event Stream
[ https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428602#comment-15428602 ] churro morales commented on HDFS-10608: --- [~surendrasingh] I will go ahead and add those fields to the AddBlockEvent you suggested [~ajisakaa] I will also remove blockPoolID as a field as it is not needed. Let me know if you think there are any other changes needed. I'll try and get an updated patch up today. > Include event for AddBlock in Inotify Event Stream > -- > > Key: HDFS-10608 > URL: https://issues.apache.org/jira/browse/HDFS-10608 > Project: Hadoop HDFS > Issue Type: Task >Reporter: churro morales >Priority: Minor > Attachments: HDFS-10608.patch, HDFS-10608.v1.patch > > > It would be nice to have an AddBlockEvent in the INotify pipeline. Based on > discussions from mailing list: > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10780) Block replication not happening on removing a volume when data being written to a datanode -- TestDataNodeHotSwapVolumes fails
Manoj Govindassamy created HDFS-10780:
-
Summary: Block replication not happening on removing a volume when
data being written to a datanode -- TestDataNodeHotSwapVolumes fails
Key: HDFS-10780
URL: https://issues.apache.org/jira/browse/HDFS-10780
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs
Affects Versions: 3.0.0-alpha1
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy
TestDataNodeHotSwapVolumes occasionally fails in the unit test
testRemoveVolumeBeingWrittenForDatanode. Data write pipeline can have issues
as there could be timeouts, data node not reachable etc, and in this test case
it was more of induced one as one of the volumes in a datanode is removed while
block write is in progress. Digging further in the logs, when the problem
happens in the write pipeline, the error recovery is not happening as expected
leading to block replication never catching up.
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 44.495 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.serv
testRemoveVolumeBeingWritten(org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes)
Time elapsed: 44.354 se
java.util.concurrent.TimeoutException: Timed out waiting for /test to reach 3
replicas
Results :
Tests in error:
TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten:637->testRemoveVolumeBeingWrittenForDatanode:714
» Timeout
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
Following exceptions are not expected in this test run
{noformat}
614 2016-08-10 12:30:11,269 [DataXceiver for client
DFSClient_NONMAPREDUCE_-640082112_10 at /127.0.0.1:58805 [Receiving block
BP-1852988604-172.16.3.66-1470857409044:blk_1073741825_1001]] DEBUG datanode.Da
taNode (DataXceiver.java:run(320)) - 127.0.0.1:58789:Number of active
connections is: 2
615 java.lang.IllegalMonitorStateException
616 at java.lang.Object.wait(Native Method)
617 at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.waitVolumeRemoved(FsVolumeList.java:280)
618 at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.removeVolumes(FsDatasetImpl.java:517)
619 at
org.apache.hadoop.hdfs.server.datanode.DataNode.removeVolumes(DataNode.java:832)
620 at
org.apache.hadoop.hdfs.server.datanode.DataNode.removeVolumes(DataNode.java:798)
{noformat}
{noformat}
720 2016-08-10 12:30:11,287 [DataNode:
[[[DISK]file:/Users/manoj/work/ups-hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
[DISK]file:/Users/manoj/work/ups-hadoop/hadoop-hdfs-projec
t/hadoop-hdfs/target/test/data/dfs/data/data2/]] heartbeating to
localhost/127.0.0.1:58788] ERROR datanode.DataNode
(BPServiceActor.java:run(768)) - Exception in BPOfferService for Block pool
BP-18529 88604-172.16.3.66-1470857409044 (Datanode Uuid
711d58ad-919d-4350-af1e-99fa0b061244) service to localhost/127.0.0.1:58788
721 java.lang.NullPointerException
722 at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1841)
723 at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:336)
724 at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:624)
725 at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:766)
726 at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-5135) Umbrella JIRA for NFS end to end unit test frameworks
[ https://issues.apache.org/jira/browse/HDFS-5135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-5135: Assignee: (was: Zhe Zhang) > Umbrella JIRA for NFS end to end unit test frameworks > - > > Key: HDFS-5135 > URL: https://issues.apache.org/jira/browse/HDFS-5135 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.2.0 >Reporter: Brandon Li > Attachments: TestRPCMessagesInNFS.java > > > Currently, we have to manually start portmap and nfs3 processes to test patch > and new functionalities. This JIRA is to track the effort to introduce a test > framework to NFS unit test without starting standalone nfs3 processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9392) Admins support for maintenance state
[
https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9392:
--
Attachment: HDFS-9392.patch
Here is the draft patch. It also includes datanode state machine transition
under some scenarios and part of HDFS-9388 to reuse test code. The complete
state machine transition such as ENTERING_MAINTENANCE to IN_MAINTENANCE and the
block management will be handled by HDFS-9390.
* Add maintenance support to the json based configuration.
* If the DN is {{DECOMMISSIONED}} or {{DEAD}}, transition it to
{{IN_MAINTENANCE}}.
* If the DN is {{NORMAL, Live}} or {{DECOMMISSION_IN_PROGRESS, Live}},
transition it to {{ENTERING_MAINTENANCE, Live}}.
* Upon maintenance expiration, transition it to {{NORMAL}} state.
> Admins support for maintenance state
>
>
> Key: HDFS-9392
> URL: https://issues.apache.org/jira/browse/HDFS-9392
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ming Ma
> Attachments: HDFS-9392.patch
>
>
> This is to allow admins to put nodes into maintenance state with optional
> timeout value as well as take nodes out of maintenance state. Likely we will
> leverage what we come up in https://issues.apache.org/jira/browse/HDFS-9005.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10779) Rename does not need to re-solve destination
Daryn Sharp created HDFS-10779:
--
Summary: Rename does not need to re-solve destination
Key: HDFS-10779
URL: https://issues.apache.org/jira/browse/HDFS-10779
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Rename uses {{FSDirectory.isDir(String)}} to determine if the destination is a
directory. This dissect the path, creates an IIP, checks if the last inode is
a directory. The rename operations already have the IIP and can check it
directly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-9632) libhdfs++: Add additional type-safe getters to the Configuration class
[ https://issues.apache.org/jira/browse/HDFS-9632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer resolved HDFS-9632. --- Resolution: Won't Fix Marking won't fix. Many of the getters included in the patch have already been implemented elsewhere. Adding on an as-needed basis has been working well so far. If we need to drop a large new set in or significantly change the API we can reopen this and do them in bulk. > libhdfs++: Add additional type-safe getters to the Configuration class > -- > > Key: HDFS-9632 > URL: https://issues.apache.org/jira/browse/HDFS-9632 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen > Attachments: HDFS-9632.HDFS-8707.000.patch > > > Notably, URIs and byte sizes are missing -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[
https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428291#comment-15428291
]
Hadoop QA commented on HDFS-8905:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 130
unchanged - 15 fixed = 130 total (was 145) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824561/HDFS-8905-v12.patch |
| JIRA Issue | HDFS-8905 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7fcf10bf5732 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 091dd19 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16481/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16481/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16481/testReport/ |
[jira] [Updated] (HDFS-10711) Optimize FSPermissionChecker group membership check
[ https://issues.apache.org/jira/browse/HDFS-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10711: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.0.0-alpha2 2.9.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. > Optimize FSPermissionChecker group membership check > --- > > Key: HDFS-10711 > URL: https://issues.apache.org/jira/browse/HDFS-10711 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HDFS-10711.1.patch, HDFS-10711.patch > > > HADOOP-13442 obviates the need for multiple group related object allocations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10711) Optimize FSPermissionChecker group membership check
[ https://issues.apache.org/jira/browse/HDFS-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428278#comment-15428278 ] Hudson commented on HDFS-10711: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10309 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10309/]) HDFS-10711. Optimize FSPermissionChecker group membership check. (kihwal: rev 2550371f66c49fe0e40aadaa68744311270084ce) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSPermissionChecker.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAttrOp.java > Optimize FSPermissionChecker group membership check > --- > > Key: HDFS-10711 > URL: https://issues.apache.org/jira/browse/HDFS-10711 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10711.1.patch, HDFS-10711.patch > > > HADOOP-13442 obviates the need for multiple group related object allocations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10711) Optimize FSPermissionChecker group membership check
[
https://issues.apache.org/jira/browse/HDFS-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428240#comment-15428240
]
Kihwal Lee commented on HDFS-10711:
---
and also the test passes fine.
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 87.241 sec
- in org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
Results :
Tests run: 28, Failures: 0, Errors: 0, Skipped: 0
{noformat}
> Optimize FSPermissionChecker group membership check
> ---
>
> Key: HDFS-10711
> URL: https://issues.apache.org/jira/browse/HDFS-10711
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10711.1.patch, HDFS-10711.patch
>
>
> HADOOP-13442 obviates the need for multiple group related object allocations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10711) Optimize FSPermissionChecker group membership check
[ https://issues.apache.org/jira/browse/HDFS-10711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428219#comment-15428219 ] Kihwal Lee commented on HDFS-10711: --- +1 The change looks good. > Optimize FSPermissionChecker group membership check > --- > > Key: HDFS-10711 > URL: https://issues.apache.org/jira/browse/HDFS-10711 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10711.1.patch, HDFS-10711.patch > > > HADOOP-13442 obviates the need for multiple group related object allocations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10768) Optimize mkdir ops
[
https://issues.apache.org/jira/browse/HDFS-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428198#comment-15428198
]
Hadoop QA commented on HDFS-10768:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} HDFS-10768 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824008/HDFS-10768.patch |
| JIRA Issue | HDFS-10768 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16482/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize mkdir ops
> --
>
> Key: HDFS-10768
> URL: https://issues.apache.org/jira/browse/HDFS-10768
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10768.patch
>
>
> Directory creation causes excessive object allocation: ex. an immutable list
> builder, containing the string of components converted from the IIP's
> byte[]s, sublist views of the string list, iterable, followed by string to
> byte[] conversion. This can all be eliminated by accessing the component's
> byte[] in the IIP.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10768) Optimize mkdir ops
[ https://issues.apache.org/jira/browse/HDFS-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10768: -- Status: Patch Available (was: Open) > Optimize mkdir ops > -- > > Key: HDFS-10768 > URL: https://issues.apache.org/jira/browse/HDFS-10768 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10768.patch > > > Directory creation causes excessive object allocation: ex. an immutable list > builder, containing the string of components converted from the IIP's > byte[]s, sublist views of the string list, iterable, followed by string to > byte[] conversion. This can all be eliminated by accessing the component's > byte[] in the IIP. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-8905: Attachment: HDFS-8905-v12.patch fix check style reported issues > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v2.patch, > HDFS-8905-v3.patch, HDFS-8905-v4.patch, HDFS-8905-v5.patch, > HDFS-8905-v6.patch, HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[
https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428132#comment-15428132
]
Hadoop QA commented on HDFS-8905:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
130 unchanged - 15 fixed = 132 total (was 145) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestMissingBlocksAlert |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824550/HDFS-8905-v11.patch |
| JIRA Issue | HDFS-8905 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 31e20c7c1dc6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8aed3741 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16480/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16480/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16480/testReport/ |
| modules | C:
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15428006#comment-15428006
]
Yiqun Lin commented on HDFS-10778:
--
Thanks [~ajisakaa] for the quick response. I will attach a new patch to address
your comment next week.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427992#comment-15427992 ] SammiChen commented on HDFS-8905: - fix the issue found in unit test failure > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-8905: Attachment: HDFS-8905-v11.patch > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427867#comment-15427867 ] Yuanbo Liu commented on HDFS-10645: --- [~ajisakaa] It's very kind of you to rebase the patch for branch-2. Thanks very much ! > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10645-branch-2.001.patch, HDFS-10645.001.patch, > HDFS-10645.002.patch, HDFS-10645.003.patch, HDFS-10645.004.patch, > HDFS-10645.005.patch, HDFS-10645.006.patch, HDFS-10645.007.patch, > HDFS-10645.008.patch, HDFS-10645.009.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10608) Include event for AddBlock in Inotify Event Stream
[ https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427805#comment-15427805 ] Akira Ajisaka commented on HDFS-10608: -- I'm thinking block pool id is not important. A block pool id is unchanged in a single NameNode, so it can be removed. > Include event for AddBlock in Inotify Event Stream > -- > > Key: HDFS-10608 > URL: https://issues.apache.org/jira/browse/HDFS-10608 > Project: Hadoop HDFS > Issue Type: Task >Reporter: churro morales >Priority: Minor > Attachments: HDFS-10608.patch, HDFS-10608.v1.patch > > > It would be nice to have an AddBlockEvent in the INotify pipeline. Based on > discussions from mailing list: > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1542#comment-1542 ] Hudson commented on HDFS-10645: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10305 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10305/]) HDFS-10645. Make block report size as a metric and add this metric to (aajisaka: rev 8179f9a493c1b26deb6b1bffacd6a829586b7f98) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java * (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/datanode.html * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10645-branch-2.001.patch, HDFS-10645.001.patch, > HDFS-10645.002.patch, HDFS-10645.003.patch, HDFS-10645.004.patch, > HDFS-10645.005.patch, HDFS-10645.006.patch, HDFS-10645.007.patch, > HDFS-10645.008.patch, HDFS-10645.009.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427774#comment-15427774
]
Akira Ajisaka commented on HDFS-10778:
--
Thanks [~linyiqun] for the patch. The improved output looks great.
Changing the output format of CLI is incompatible, so would you add a new
option to optimize the output? '-h' is good for me. In addition, we need to
document the option.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427773#comment-15427773
]
Jingcheng Du commented on HDFS-9668:
[~fenghua_hu], thanks for the comments!
The new patch moves the disk operations out of lock scope, and make it rely on
locks in file system which might probably lead to race conditions and
inconsistency in both memory map and files. Like what I mentioned above, it
might lead to inconsistency when creating replicas and removing volumes happen
concurrently.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-2.patch, HDFS-9668-3.patch,
> execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can implement the lock by either
[jira] [Issue Comment Deleted] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-8905: Comment: was deleted (was: rebase the patch on latest trunk code ) > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v2.patch, HDFS-8905-v3.patch, HDFS-8905-v4.patch, > HDFS-8905-v5.patch, HDFS-8905-v6.patch, HDFS-8905-v7.patch, > HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-8905: Attachment: (was: HDFS-8905-v10.patch) > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v2.patch, HDFS-8905-v3.patch, HDFS-8905-v4.patch, > HDFS-8905-v5.patch, HDFS-8905-v6.patch, HDFS-8905-v7.patch, > HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-8905: Attachment: HDFS-8905-v10.patch rebase the patch on latest trunk code > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v10.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10645: - Fix Version/s: 3.0.0-alpha2 > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10645-branch-2.001.patch, HDFS-10645.001.patch, > HDFS-10645.002.patch, HDFS-10645.003.patch, HDFS-10645.004.patch, > HDFS-10645.005.patch, HDFS-10645.006.patch, HDFS-10645.007.patch, > HDFS-10645.008.patch, HDFS-10645.009.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HDFS-10645: - Attachment: HDFS-10645-branch-2.001.patch Committed this to trunk. I had to rebase the patch for branch-2, so attaching a patch for branch-2. I'll commit this to branch-2 if Jenkins says okay. > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Attachments: HDFS-10645-branch-2.001.patch, HDFS-10645.001.patch, > HDFS-10645.002.patch, HDFS-10645.003.patch, HDFS-10645.004.patch, > HDFS-10645.005.patch, HDFS-10645.006.patch, HDFS-10645.007.patch, > HDFS-10645.008.patch, HDFS-10645.009.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427750#comment-15427750
]
Jingcheng Du commented on HDFS-9668:
Thanks [~arpitagarwal] for the comments.
I agree, this new patch introduces more risks, and I plan to go back to my old
patches (which uses locks) and refine it to address the concerns in this JIRA.
Thanks a lot.
HDFS-10682 had changed the synchronized to a separate lock, and I can do my
refinement on the old patches more easily. Thanks.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-2.patch, HDFS-9668-3.patch,
> execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can implement the lock by either storage level or
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427691#comment-15427691
]
Fenghua Hu commented on HDFS-9668:
--
[~jingcheng...@intel.com]，
We reviewed your lock patch, but didn't find any apparent issues, except that
some external references to FsDatasetImpl object needs to be modified
correspondingly. "I realize that the patch is not implemented properly.", Could
you please elaborate your concern so that we can think about again? Thanks.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-2.patch, HDFS-9668-3.patch,
> execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can implement the lock by either storage level or block-level.
[jira] [Commented] (HDFS-9668) Optimize the locking in FsDatasetImpl
[
https://issues.apache.org/jira/browse/HDFS-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427678#comment-15427678
]
Arpit Agarwal commented on HDFS-9668:
-
Hi all, HDFS-10682 recently replaced the FsDatasetImpl lock with a separate
Re-entrant lock. The goal is to make it easier to replace it with an
instrumented lock (HDFS-10742) or a read-write lock, eventually.
Moving expensive operations out of the lock is a good goal but I still think it
will be a risky change. Using a read-write lock is an easier fix but even that
needs care.
> Optimize the locking in FsDatasetImpl
> -
>
> Key: HDFS-9668
> URL: https://issues.apache.org/jira/browse/HDFS-9668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HDFS-9668-1.patch, HDFS-9668-2.patch, HDFS-9668-3.patch,
> execution_time.png
>
>
> During the HBase test on a tiered storage of HDFS (WAL is stored in
> SSD/RAMDISK, and all other files are stored in HDD), we observe many
> long-time BLOCKED threads on FsDatasetImpl in DataNode. The following is part
> of the jstack result:
> {noformat}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48521 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779272_40852]" - Thread
> t@93336
>java.lang.Thread.State: BLOCKED
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:)
> - waiting to lock <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl) owned by
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" t@93335
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
>
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1626037897_1 at
> /192.168.50.16:48520 [Receiving block
> BP-1042877462-192.168.50.13-1446173170517:blk_1073779271_40851]" - Thread
> t@93335
>java.lang.Thread.State: RUNNABLE
> at java.io.UnixFileSystem.createFileExclusively(Native Method)
> at java.io.File.createNewFile(File.java:1012)
> at
> org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createTmpFile(DatanodeUtil.java:66)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:271)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:286)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1140)
> - locked <18324c9> (a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
> at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:113)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:183)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:615)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:235)
> at java.lang.Thread.run(Thread.java:745)
>Locked ownable synchronizers:
> - None
> {noformat}
> We measured the execution of some operations in FsDatasetImpl during the
> test. Here following is the result.
> !execution_time.png!
> The operations of finalizeBlock, addBlock and createRbw on HDD in a heavy
> load take a really long time.
> It means one slow operation of finalizeBlock, addBlock and createRbw in a
> slow storage can block all the other same operations in the same DataNode,
> especially in HBase when many wal/flusher/compactor are configured.
> We need a finer grained lock mechanism in a new FsDatasetImpl implementation
> and users can choose the implementation by configuring
> "dfs.datanode.fsdataset.factory" in DataNode.
> We can
