[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432192#comment-15432192 ] Kai Zheng commented on HDFS-8905: - Thanks Sammi. I will commit this by end of tomorrow if no more comments or concerns. > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: SammiChen > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v13.patch, > HDFS-8905-v14.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432184#comment-15432184
]
Akira Ajisaka commented on HDFS-10783:
--
Thanks [~linyiqun] for updating the patch.
Would you fix the checkstyle warning? I'm +1 if that is addressed.
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -maxSize
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432179#comment-15432179 ] SammiChen commented on HDFS-8905: - I run the failed unit cases locally with the latest HDFS trunk + this patch, four unit cases passed. But TestDFSUpgradeWithHA still failed. I again run the TestDFSUpgradeWithHA against the latest HDFS trunk code, it failed too. And I have double checked the failed unit case, it's not related with the code change in the patch. > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: SammiChen > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v13.patch, > HDFS-8905-v14.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9392) Admins support for maintenance state
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ming Ma updated HDFS-9392: -- Attachment: HDFS-9392-3.patch Thanks [~eddyxu]! Here is the new patch with the checkstyle fixes. > Admins support for maintenance state > > > Key: HDFS-9392 > URL: https://issues.apache.org/jira/browse/HDFS-9392 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ming Ma > Attachments: HDFS-9392-2.patch, HDFS-9392-3.patch, HDFS-9392.patch > > > This is to allow admins to put nodes into maintenance state with optional > timeout value as well as take nodes out of maintenance state. Likely we will > leverage what we come up in https://issues.apache.org/jira/browse/HDFS-9005. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10340) data node sudden killed
[ https://issues.apache.org/jira/browse/HDFS-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432154#comment-15432154 ] KWON BYUNGCHANG commented on HDFS-10340: if user of DataNode is same with NodeManager, maybe it is related with YARN-4459 > data node sudden killed > > > Key: HDFS-10340 > URL: https://issues.apache.org/jira/browse/HDFS-10340 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.6.0 > Environment: Ubuntu 16.04 LTS , RAM 16g , cpu core : 8 , hdd 100gb, > hadoop 2.6.0 >Reporter: tu nguyen khac >Priority: Critical > > I tried to setup a new data node using ubuntu 16 > and get it join to an existed Hadoop Hdfs cluster ( there are 10 nodes in > this cluster and they all run on centos Os 6 ) > But when i try to boostrap this node , after about 10 or 20 minutes i get > this strange errors : > 2016-04-26 20:12:09,394 INFO > org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: > /10.3.24.65:55323, dest: /10.3.24.197:50010, bytes: 79902, op: HDFS_WRITE, > cliID: DFSClient_NONMAPREDUCE_1379996362_1, offset: 0, srvID: > 225f5b43-1dd3-4ac6-88d2-1e8d27dba55b, blockid: > BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, duration: > 15331628 > 2016-04-26 20:12:09,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > PacketResponder: BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, > type=LAST_IN_PIPELINE, downstreams=0:[] terminating > 2016-04-26 20:12:25,410 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038502_789829 > 2016-04-26 20:12:25,411 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832 > 2016-04-26 20:13:18,546 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: > Scheduling blk_1074038502_789829 file > /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502 > for deletion > 2016-04-26 20:13:18,562 INFO > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: > Deleted BP-352432948-10.3.24.65-1433821675295 blk_1074038502_789829 file > /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502 > 2016-04-26 20:15:46,481 ERROR > org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM > 2016-04-26 20:15:46,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: > SHUTDOWN_MSG: > / > SHUTDOWN_MSG: Shutting down DataNode at bigdata-dw-24-197/10.3.24.197 > / -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10756) Expose getTrashRoot to HTTPFS and WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432139#comment-15432139
]
Hadoop QA commented on HDFS-10756:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 8m 45s{color}
| {color:red} root generated 1 new + 708 unchanged - 1 fixed = 709 total (was
709) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 57s{color} | {color:orange} root: The patch generated 3 new + 941 unchanged
- 0 fixed = 944 total (was 941) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
8s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 37s{color}
| {color:red} hadoop-hdfs-httpfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}152m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.fs.http.server.TestHttpFSServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824723/HDFS-10756.001.patch |
| JIRA Issue | HDFS-10756 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b7efb030e82c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c49333b |
| Default Java | 1.8.0_101 |
| findbugs |
[jira] [Commented] (HDFS-10756) Expose getTrashRoot to HTTPFS and WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432121#comment-15432121
]
Xiao Chen commented on HDFS-10756:
--
Thank you for working on this, [~yuanbo]. Idea looks good.
I don't have a strong feeling regarding splitting the jira. If we're splitting
though, I'd prefer we have just 2 jiras for webhdfs and httpfs respectively,
and both include their own doc changes.
Some comments:
* In both implementations' {{getTrashRoot}}:
** I see we return null when an IOE is caught. This is not the same behavior as
dfs, which always returns {{super.getTrashRoot(path);}}.
** {{LOG.warn("can not find trash root of " + p);}}: Since we're using slf4j,
let's use place holders when logging. Also, suggest s/can not/Cannot/g here.
* FSOperations.java
** Suggest we keep the whitespace as-is, to minimize our change.
** {{@InterfaceAudience.Private}} should be on a separate line on its own.
* NamenodeWebHdfsMethods.java
** {{getTrashRoot}}'s inner comment is not accurate. {{queryPath.isRoot() ==
true}} doesn't mean it is not an EZ. If {{/}} is an EZ, we choose to user's
home dir because it's under {{/}} as well. How about we add a javadoc to the
method, and remove the comment here?
* Test
** Can we add a case to verify nested ez's trash dir works too?
* Please fix the checkstyles when jenkins comes back.
> Expose getTrashRoot to HTTPFS and WebHDFS
> -
>
> Key: HDFS-10756
> URL: https://issues.apache.org/jira/browse/HDFS-10756
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: encryption, httpfs, webhdfs
>Reporter: Xiao Chen
>Assignee: Yuanbo Liu
> Attachments: HDFS-10756.001.patch
>
>
> Currently, hadoop FileSystem API has
> [getTrashRoot|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2708]
> to determine trash directory at run time. Default trash dir is under
> {{/user/$USER}}
> For an encrypted file, since moving files between/in/out of EZs are not
> allowed, when an EZ file is deleted via CLI, it calls in to [DFS
> implementation|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2485]
> to move the file to a trash directory under the same EZ.
> This works perfectly fine for CLI users or java users who call FileSystem
> API. But for users via httpfs/webhdfs, currently there is no way to figure
> out what the trash root would be. This jira is proposing we add such
> interface to httpfs and webhdfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[
https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432111#comment-15432111
]
Hadoop QA commented on HDFS-8905:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 130
unchanged - 15 fixed = 130 total (was 145) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.TestRenameWhileOpen |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824974/HDFS-8905-v14.patch |
| JIRA Issue | HDFS-8905 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3d0439c71020 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c49333b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16505/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-10784) Implement WebHdfsFileSystem#listStatusIterator
[
https://issues.apache.org/jira/browse/HDFS-10784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432087#comment-15432087
]
Hadoop QA commented on HDFS-10784:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 21s{color}
| {color:red} hadoop-hdfs-project generated 1 new + 51 unchanged - 1 fixed = 52
total (was 52) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project: The patch generated 10 new
+ 252 unchanged - 3 fixed = 262 total (was 255) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.mover.TestStorageMover |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824969/HDFS-10784.001.patch |
| JIRA Issue | HDFS-10784 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 24f0506a7f26 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c49333b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/diff-compile-javac-hadoop-hdfs-project.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16503/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
|
| unit |
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432076#comment-15432076 ] Kai Zheng commented on HDFS-8905: - Thanks Sammi for the update. The latest change looks good to me, in addition to the earlier changes being reviewed already. +1 pending on Jenkins. > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: SammiChen > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v13.patch, > HDFS-8905-v14.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[ https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SammiChen updated HDFS-8905: Attachment: HDFS-8905-v14.patch Thanks for Kai's review and suggestion. I remove the unnecessary synchronized statements and add comments for readFromBuffer function about is parameter src 's usage. > Refactor DFSInputStream#ReaderStrategy > -- > > Key: HDFS-8905 > URL: https://issues.apache.org/jira/browse/HDFS-8905 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Kai Zheng >Assignee: SammiChen > Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch, > HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v13.patch, > HDFS-8905-v14.patch, HDFS-8905-v2.patch, HDFS-8905-v3.patch, > HDFS-8905-v4.patch, HDFS-8905-v5.patch, HDFS-8905-v6.patch, > HDFS-8905-v7.patch, HDFS-8905-v8.patch, HDFS-8905-v9.patch > > > DFSInputStream#ReaderStrategy family don't look very good. This refactors a > little bit to make them make more sense. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10756) Expose getTrashRoot to HTTPFS and WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuanbo Liu updated HDFS-10756:
--
Status: Patch Available (was: Open)
> Expose getTrashRoot to HTTPFS and WebHDFS
> -
>
> Key: HDFS-10756
> URL: https://issues.apache.org/jira/browse/HDFS-10756
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: encryption, httpfs, webhdfs
>Reporter: Xiao Chen
>Assignee: Yuanbo Liu
> Attachments: HDFS-10756.001.patch
>
>
> Currently, hadoop FileSystem API has
> [getTrashRoot|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2708]
> to determine trash directory at run time. Default trash dir is under
> {{/user/$USER}}
> For an encrypted file, since moving files between/in/out of EZs are not
> allowed, when an EZ file is deleted via CLI, it calls in to [DFS
> implementation|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2485]
> to move the file to a trash directory under the same EZ.
> This works perfectly fine for CLI users or java users who call FileSystem
> API. But for users via httpfs/webhdfs, currently there is no way to figure
> out what the trash root would be. This jira is proposing we add such
> interface to httpfs and webhdfs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10645) Make block report size as a metric and add this metric to datanode web ui
[ https://issues.apache.org/jira/browse/HDFS-10645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431990#comment-15431990 ] Yuanbo Liu commented on HDFS-10645: --- [~ajisakaa] Thanks for your committing! > Make block report size as a metric and add this metric to datanode web ui > - > > Key: HDFS-10645 > URL: https://issues.apache.org/jira/browse/HDFS-10645 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, ui >Reporter: Yuanbo Liu >Assignee: Yuanbo Liu > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: HDFS-10645-branch-2.001.patch, HDFS-10645.001.patch, > HDFS-10645.002.patch, HDFS-10645.003.patch, HDFS-10645.004.patch, > HDFS-10645.005.patch, HDFS-10645.006.patch, HDFS-10645.007.patch, > HDFS-10645.008.patch, HDFS-10645.009.patch, Selection_047.png, > Selection_048.png > > > Record block report size as a metric and show it on datanode UI. It's > important for administrators to know the bottleneck of block report, and the > metric is also a good tuning metric. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8312) Trash does not descent into child directories to check for permissions
[
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431978#comment-15431978
]
Hudson commented on HDFS-8312:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10323 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10323/])
HDFS-8312. Added permission check for moving file to Trash. (Weiwei Yang
(eyang: rev c49333becfa7652460976a61eb86522010bcfeed)
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/TrashPolicyDefault.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/ClientNamenodeProtocol.proto
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSPermission.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirRenameOp.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java
* (edit)
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java
> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, security
>Affects Versions: 2.2.0, 2.6.0, 2.7.2
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: HDFS-8312-001.patch, HDFS-8312-002.patch,
> HDFS-8312-003.patch, HDFS-8312-004.patch, HDFS-8312-005.patch,
> HDFS-8312-testcase.patch
>
>
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2.
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at:
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL,
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior. When
> trash is enabled, files owned by user1 is deleted by user2. It looks like
> trash does not recursively validate if the child directory files can be
> removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10784) Implement WebHdfsFileSystem#listStatusIterator
[ https://issues.apache.org/jira/browse/HDFS-10784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10784: --- Attachment: HDFS-10784.001.patch Patch attached. I couldn't find a nice way of codesharing with the other iterator implementations, but it isn't that much. > Implement WebHdfsFileSystem#listStatusIterator > -- > > Key: HDFS-10784 > URL: https://issues.apache.org/jira/browse/HDFS-10784 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-10784.001.patch > > > It would be nice to implement the iterative listStatus in WebHDFS so client > apps do not need to buffer the full file list for large directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10784) Implement WebHdfsFileSystem#listStatusIterator
[ https://issues.apache.org/jira/browse/HDFS-10784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10784: --- Status: Patch Available (was: Open) > Implement WebHdfsFileSystem#listStatusIterator > -- > > Key: HDFS-10784 > URL: https://issues.apache.org/jira/browse/HDFS-10784 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 2.6.4 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-10784.001.patch > > > It would be nice to implement the iterative listStatus in WebHDFS so client > apps do not need to buffer the full file list for large directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10784) Implement WebHdfsFileSystem#listStatusIterator
Andrew Wang created HDFS-10784: -- Summary: Implement WebHdfsFileSystem#listStatusIterator Key: HDFS-10784 URL: https://issues.apache.org/jira/browse/HDFS-10784 Project: Hadoop HDFS Issue Type: Improvement Components: webhdfs Affects Versions: 2.6.4 Reporter: Andrew Wang Assignee: Andrew Wang It would be nice to implement the iterative listStatus in WebHDFS so client apps do not need to buffer the full file list for large directories. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8312) Trash does not descent into child directories to check for permissions
[
https://issues.apache.org/jira/browse/HDFS-8312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-8312:
Resolution: Fixed
Fix Version/s: 2.9.0
Release Note: HDFS-8312. Added permission check for moving file to Trash.
(Weiwei Yang via Eric Yang)
Status: Resolved (was: Patch Available)
+1 looks good. I just committed this to branch-2 and trunk. Thank you, Weiwei.
> Trash does not descent into child directories to check for permissions
> --
>
> Key: HDFS-8312
> URL: https://issues.apache.org/jira/browse/HDFS-8312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, security
>Affects Versions: 2.2.0, 2.6.0, 2.7.2
>Reporter: Eric Yang
>Assignee: Weiwei Yang
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: HDFS-8312-001.patch, HDFS-8312-002.patch,
> HDFS-8312-003.patch, HDFS-8312-004.patch, HDFS-8312-005.patch,
> HDFS-8312-testcase.patch
>
>
> HDFS trash does not descent into child directory to check if user has
> permission to delete files. For example:
> Run the following command to initialize directory structure as super user:
> {code}
> hadoop fs -mkdir /BSS/level1
> hadoop fs -mkdir /BSS/level1/level2
> hadoop fs -mkdir /BSS/level1/level2/level3
> hadoop fs -put /tmp/appConfig.json /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown user1:users /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chown -R user1:users /BSS/level1
> hadoop fs -chown -R 750 /BSS/level1
> hadoop fs -chmod -R 640 /BSS/level1/level2/level3/testfile.txt
> hadoop fs -chmod 775 /BSS
> {code}
> Change to a normal user called user2.
> When trash is enabled:
> {code}
> sudo su user2 -
> hadoop fs -rm -r /BSS/level1
> 15/05/01 16:51:20 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 3600 minutes, Emptier interval = 0 minutes.
> Moved: 'hdfs://bdvs323.svl.ibm.com:9000/BSS/level1' to trash at:
> hdfs://bdvs323.svl.ibm.com:9000/user/user2/.Trash/Current
> {code}
> When trash is disabled:
> {code}
> /opt/ibm/biginsights/IHC/bin/hadoop fs -Dfs.trash.interval=0 -rm -r
> /BSS/level1
> 15/05/01 16:58:31 INFO fs.TrashPolicyDefault: Namenode trash configuration:
> Deletion interval = 0 minutes, Emptier interval = 0 minutes.
> rm: Permission denied: user=user2, access=ALL,
> inode="/BSS/level1":user1:users:drwxr-x---
> {code}
> There is inconsistency between trash behavior and delete behavior. When
> trash is enabled, files owned by user1 is deleted by user2. It looks like
> trash does not recursively validate if the child directory files can be
> removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9392) Admins support for maintenance state
[ https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431915#comment-15431915 ] Lei (Eddy) Xu commented on HDFS-9392: - Hi, [~mingma] Thanks for providing the patch. It looks good to me. +1 pending the checkstyle fixes. > Admins support for maintenance state > > > Key: HDFS-9392 > URL: https://issues.apache.org/jira/browse/HDFS-9392 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ming Ma > Attachments: HDFS-9392-2.patch, HDFS-9392.patch > > > This is to allow admins to put nodes into maintenance state with optional > timeout value as well as take nodes out of maintenance state. Likely we will > leverage what we come up in https://issues.apache.org/jira/browse/HDFS-9005. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9395) Make HDFS audit logging consistant
[ https://issues.apache.org/jira/browse/HDFS-9395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431749#comment-15431749 ] Ray Chiang commented on HDFS-9395: -- Also, can the new behavior be documented in the "Release Notes" section for Hadoop 3? This is only one of a few incompatible changes missing such documentation. > Make HDFS audit logging consistant > -- > > Key: HDFS-9395 > URL: https://issues.apache.org/jira/browse/HDFS-9395 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kuhu Shukla > Fix For: 3.0.0-alpha1 > > Attachments: HDFS-9395-branch-2.7.001.patch, > HDFS-9395-branch-2.7.002.patch, HDFS-9395.001.patch, HDFS-9395.002.patch, > HDFS-9395.003.patch, HDFS-9395.004.patch, HDFS-9395.005.patch, > HDFS-9395.006.patch, HDFS-9395.007.patch > > > So, the big question here is what should go in the audit log? All failures, > or just "permission denied" failures? Or, to put it a different way, if > someone attempts to do something and it fails because a file doesn't exist, > is that worth an audit log entry? > We are currently inconsistent on this point. For example, concat, > getContentSummary, addCacheDirective, and setErasureEncodingPolicy create an > audit log entry for all failures, but setOwner, delete, and setAclEntries > attempt to only create an entry for AccessControlException-based failures. > There are a few operations, like allowSnapshot, disallowSnapshot, and > startRollingUpgrade that never create audit log failure entries at all. They > simply log nothing for any failure, and log success for a successful > operation. > So to summarize, different HDFS operations currently fall into 3 categories: > 1. audit-log all failures > 2. audit-log only AccessControlException failures > 3. never audit-log failures > Which category is right? And how can we fix the inconsistency -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission
[
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431747#comment-15431747
]
Ray Chiang commented on HDFS-10650:
---
Sorry to come in late. Can the old versus new behavior be documented in the
"Release Notes" section?
> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10650.001.patch, HDFS-10650.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a
> directory.
> {code:java}
> public boolean mkdirs(String src, FsPermission permission,
> boolean createParent) throws IOException {
> if (permission == null) {
> permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
> public boolean primitiveMkdir(String src, FsPermission absPermission,
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
> absPermission =
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10178) Permanent write failures can happen if pipeline recoveries occur for the first packet
[ https://issues.apache.org/jira/browse/HDFS-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated HDFS-10178: Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3) > Permanent write failures can happen if pipeline recoveries occur for the > first packet > - > > Key: HDFS-10178 > URL: https://issues.apache.org/jira/browse/HDFS-10178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.3 > > Attachments: HDFS-10178.patch, HDFS-10178.v2.patch, > HDFS-10178.v3.patch, HDFS-10178.v4.patch, HDFS-10178.v5.patch > > > We have observed that write fails permanently if the first packet doesn't go > through properly and pipeline recovery happens. If the write op creates a > pipeline, but the actual data packet does not reach one or more datanodes in > time, the pipeline recovery will be done against the 0-byte partial block. > If additional datanodes are added, the block is transferred to the new nodes. > After the transfer, each node will have a meta file containing the header > and 0-length data block file. The pipeline recovery seems to work correctly > up to this point, but write fails when actual data packet is resent. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10178) Permanent write failures can happen if pipeline recoveries occur for the first packet
[ https://issues.apache.org/jira/browse/HDFS-10178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431746#comment-15431746 ] Chris Trezzo commented on HDFS-10178: - Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks! > Permanent write failures can happen if pipeline recoveries occur for the > first packet > - > > Key: HDFS-10178 > URL: https://issues.apache.org/jira/browse/HDFS-10178 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Critical > Fix For: 2.7.3 > > Attachments: HDFS-10178.patch, HDFS-10178.v2.patch, > HDFS-10178.v3.patch, HDFS-10178.v4.patch, HDFS-10178.v5.patch > > > We have observed that write fails permanently if the first packet doesn't go > through properly and pipeline recovery happens. If the write op creates a > pipeline, but the actual data packet does not reach one or more datanodes in > time, the pipeline recovery will be done against the 0-byte partial block. > If additional datanodes are added, the block is transferred to the new nodes. > After the transfer, each node will have a meta file containing the header > and 0-length data block file. The pipeline recovery seems to work correctly > up to this point, but write fails when actual data packet is resent. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod
[
https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431734#comment-15431734
]
Hadoop QA commented on HDFS-10754:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
51s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
27s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
46s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
48s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
3s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
4s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
7s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
19s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824924/HDFS-10754.HDFS-8707.008.patch
|
| JIRA Issue | HDFS-10754 |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
mvninstall |
| uname | Linux bfa54cf9760a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 4a74bc4 |
| Default Java | 1.7.0_111 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 |
| JDK v1.7.0_111 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16502/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods
[ https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431709#comment-15431709 ] Hudson commented on HDFS-10762: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10321 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10321/]) Revert "HDFS-10762. Pass IIP for file status related methods. (kihwal: rev 3ca4d6ddfd199c95677721ff3bcb95d1da45bd88) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java > Pass IIP for file status related methods > > > Key: HDFS-10762 > URL: https://issues.apache.org/jira/browse/HDFS-10762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10762.patch > > > The frequently called file status methods will not require path re-resolves > if the IIP is passed down the call stack. The code can be simplified further > if the IIP tracks if the original path was a reserved raw path. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods
[ https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431684#comment-15431684 ] Kihwal Lee commented on HDFS-10762: --- Although it is functionally correct, it looks like the patch undid a prior optimization. I will revert it from trunk. > Pass IIP for file status related methods > > > Key: HDFS-10762 > URL: https://issues.apache.org/jira/browse/HDFS-10762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10762.patch > > > The frequently called file status methods will not require path re-resolves > if the IIP is passed down the call stack. The code can be simplified further > if the IIP tracks if the original path was a reserved raw path. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9669) TcpPeerServer should respect ipc.server.listen.queue.size
[
https://issues.apache.org/jira/browse/HDFS-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431672#comment-15431672
]
Chris Trezzo commented on HDFS-9669:
Adding 2.6.5 to the target versions with the intention of backporting this to
branch-2.6. Please let me know if you think otherwise. Thanks!
> TcpPeerServer should respect ipc.server.listen.queue.size
> -
>
> Key: HDFS-9669
> URL: https://issues.apache.org/jira/browse/HDFS-9669
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.7.3
>
> Attachments: HDFS-9669.0.patch, HDFS-9669.1.patch, HDFS-9669.1.patch
>
>
> On periods of high traffic we are seeing:
> {code}
> 16/01/19 23:40:40 WARN hdfs.DFSClient: Connection failure: Failed to connect
> to /10.138.178.47:50010 for file /MYPATH/MYFILE for block
> BP-1935559084-10.138.112.27-1449689748174:blk_1080898601_7375294:java.io.IOException:
> Connection reset by peer
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:109)
> at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
> {code}
> At the time that this happens there are way less xceivers than configured.
> On most JDK's this will make 50 the total backlog at any time. This
> effectively means that any GC + Busy time willl result in tcp resets.
> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/net/ServerSocket.java#l370
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9669) TcpPeerServer should respect ipc.server.listen.queue.size
[
https://issues.apache.org/jira/browse/HDFS-9669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-9669:
---
Target Version/s: 2.7.3, 2.6.5 (was: 2.7.3)
> TcpPeerServer should respect ipc.server.listen.queue.size
> -
>
> Key: HDFS-9669
> URL: https://issues.apache.org/jira/browse/HDFS-9669
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.7.2
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Fix For: 2.7.3
>
> Attachments: HDFS-9669.0.patch, HDFS-9669.1.patch, HDFS-9669.1.patch
>
>
> On periods of high traffic we are seeing:
> {code}
> 16/01/19 23:40:40 WARN hdfs.DFSClient: Connection failure: Failed to connect
> to /10.138.178.47:50010 for file /MYPATH/MYFILE for block
> BP-1935559084-10.138.112.27-1449689748174:blk_1080898601_7375294:java.io.IOException:
> Connection reset by peer
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
> at sun.nio.ch.IOUtil.write(IOUtil.java:65)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:109)
> at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
> {code}
> At the time that this happens there are way less xceivers than configured.
> On most JDK's this will make 50 the total backlog at any time. This
> effectively means that any GC + Busy time willl result in tcp resets.
> http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/net/ServerSocket.java#l370
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10761) libhdfs++: Fix broken logic in HA retry policy
[ https://issues.apache.org/jira/browse/HDFS-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-10761: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review [~anatoli.shein]. I committed this to HDFS-8707. > libhdfs++: Fix broken logic in HA retry policy > -- > > Key: HDFS-10761 > URL: https://issues.apache.org/jira/browse/HDFS-10761 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer > Attachments: HDFS-10761.HDFS-8707.000.patch > > > Two issues in the HA policy: > 1) There's logic to guard against HDFS-8161, but it wasn't checking how many > failovers had already happened. It'd get stuck alternating between nodes > forever. > 2) Switched a ">" for a "<" so when max failovers was reached it'd keep > trying anyway. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431650#comment-15431650
]
Xiao Chen commented on HDFS-8986:
-
Test failure looks unrelated and passed locally.
Checkstyle warning I think can be ignored - it's more readable this way and
hadoop builders are named this way in many fundamental places like acls, rpcs,
etc.
> Add option to -du to calculate directory space usage excluding snapshots
>
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch,
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch,
> HDFS-8986.06.patch, HDFS-8986.07.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its
> children), the report includes space consumed by blocks that are only present
> in the snapshots. This is confusing for end users.
> {noformat}
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0 0 /tmp/parent
> 0 0 /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related
> disk usage in the output
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod
[ https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-10754: - Attachment: HDFS-10754.HDFS-8707.008.patch Somehow recursion counter became not atomic in the last patch. Fixed not. > libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, > hdfs_chown, and hdfs_chmod > > > Key: HDFS-10754 > URL: https://issues.apache.org/jira/browse/HDFS-10754 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-10754.HDFS-8707.000.patch, > HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, > HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, > HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, > HDFS-10754.HDFS-8707.007.patch, HDFS-10754.HDFS-8707.008.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods
[ https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431621#comment-15431621 ] Hudson commented on HDFS-10762: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10319 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10319/]) HDFS-10762. Pass IIP for file status related methods. Contributed by (kihwal: rev 22fc46d7659972ff016ccf1c6f781f0c160be26f) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReservedRawPaths.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodesInPath.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java > Pass IIP for file status related methods > > > Key: HDFS-10762 > URL: https://issues.apache.org/jira/browse/HDFS-10762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10762.patch > > > The frequently called file status methods will not require path re-resolves > if the IIP is passed down the call stack. The code can be simplified further > if the IIP tracks if the original path was a reserved raw path. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431620#comment-15431620
]
Hadoop QA commented on HDFS-8986:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
52s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 28s{color} | {color:orange} root: The patch generated 4 new + 352 unchanged
- 11 fixed = 356 total (was 363) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
32s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
12s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandby |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824903/HDFS-8986.07.patch |
| JIRA Issue | HDFS-8986 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml cc |
| uname | Linux 4d6251bf3a39 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / dc7a1c5 |
| Default Java | 1.8.0_101 |
|
[jira] [Commented] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod
[
https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431603#comment-15431603
]
Hadoop QA commented on HDFS-10754:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
42s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
45s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
2s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
6s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
8s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
9s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with
JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824913/HDFS-10754.HDFS-8707.007.patch
|
| JIRA Issue | HDFS-10754 |
| Optional Tests | asflicense compile cc mvnsite javac unit javadoc
mvninstall |
| uname | Linux 76c717d4fd6a 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / c64f612 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 |
| JDK v1.7.0_101 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16501/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
|
[jira] [Updated] (HDFS-8845) DiskChecker should not traverse the entire tree
[ https://issues.apache.org/jira/browse/HDFS-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Trezzo updated HDFS-8845: --- Target Version/s: 2.8.0, 2.6.5 (was: 2.8.0) > DiskChecker should not traverse the entire tree > --- > > Key: HDFS-8845 > URL: https://issues.apache.org/jira/browse/HDFS-8845 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.7.3 > > Attachments: HDFS-8845.patch > > > DiskChecker should not traverse entire tree because it's causing heavy disk > load on checkDiskError() -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8845) DiskChecker should not traverse the entire tree
[ https://issues.apache.org/jira/browse/HDFS-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431594#comment-15431594 ] Chris Trezzo commented on HDFS-8845: Adding 2.6.5 to the target versions with the intention of backporting this to branch-2.6. Please let me know if you think otherwise. Thanks! > DiskChecker should not traverse the entire tree > --- > > Key: HDFS-8845 > URL: https://issues.apache.org/jira/browse/HDFS-8845 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.7.3 > > Attachments: HDFS-8845.patch > > > DiskChecker should not traverse entire tree because it's causing heavy disk > load on checkDiskError() -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods
[ https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431588#comment-15431588 ] Kihwal Lee commented on HDFS-10762: --- [~daryn], please post a branch-2 version of the patch. > Pass IIP for file status related methods > > > Key: HDFS-10762 > URL: https://issues.apache.org/jira/browse/HDFS-10762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10762.patch > > > The frequently called file status methods will not require path re-resolves > if the IIP is passed down the call stack. The code can be simplified further > if the IIP tracks if the original path was a reserved raw path. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10762) Pass IIP for file status related methods
[ https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-10762: -- Fix Version/s: 3.0.0-alpha2 > Pass IIP for file status related methods > > > Key: HDFS-10762 > URL: https://issues.apache.org/jira/browse/HDFS-10762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10762.patch > > > The frequently called file status methods will not require path re-resolves > if the IIP is passed down the call stack. The code can be simplified further > if the IIP tracks if the original path was a reserved raw path. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods
[
https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431553#comment-15431553
]
Kihwal Lee commented on HDFS-10762:
---
Nice change. I am convinced that there will be no case where
{{createFileStatus()}} is called with a different snapshot ID than what is for
the path. Carrying snapshot ID and raw path check in IIP makes sense.
+1
> Pass IIP for file status related methods
>
>
> Key: HDFS-10762
> URL: https://issues.apache.org/jira/browse/HDFS-10762
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10762.patch
>
>
> The frequently called file status methods will not require path re-resolves
> if the IIP is passed down the call stack. The code can be simplified further
> if the IIP tracks if the original path was a reserved raw path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod
[ https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431511#comment-15431511 ] Anatoli Shein commented on HDFS-10754: -- Thanks for the review, [~James C]. I have addressed your comments in the new patch as follows: 1) There's a good bit of code duplication in the config parsing code for each tool. Factoring that out into a set of shared files makes each tool cleaner and also helps improve consistency in how they deal with configuration. (/) I changed the way configs are parsed, and separated them into a separate function. For examples we just now look at the configs and do not care about the given uri, and for tools we look at both. 2) Very minor, disregard if you want, but you rather than including google/protobuf/io/coded_stream.h you could include google/protobuf/stubs/common.h if you just need something to declare ShutdownProtobufLibrary. It should be a bit smaller so it'd help with compile times and brings in fewer symbols. (/) Fixed. 3) In places you call "optional::value" you can also call "optional::value_or" to get rid of some if blocks without making the code any harder to read (IMO). Nested value_or calls do tend to make things hard to read. The block below and the equivalent block for host are repeated in multiple places, perhaps make a function that takes the uri, options, and default and stick the logic there and share it between executables in the same was as the config stuff. (/) I got rid of this whole part altogether, since ConfigLoader is now doing this part. 4) One of the things that's getting more and more important is adding tests; admittedly I haven't been getting good CI coverage on my recent patches either. Could you include some simple tests, even a bash script would do, where you run these tools? One method would be to assume there's a cluster running and you're passed in something like $HADOOP_CLI (if you need to write anything), $NN_HOST, and $NN_PORT just so it's easy enough for people to run the tests. One of the major blockers we have in the way of merging this branch into trunk is getting really solid tests so if you think of a way to test these in the CI test environment that's ideal. (/) I will add tests for this when the find tool patch (HDFS-10679) lands, because I need it for the result verification. 5) Use std::make_shared rather than passing a pointer into the constructor of shared_ptr. make_shared will allocate the reference count variable contiguously with the rest of the class so you get less heap fragmentation and better locality. It's also exception safe which is always a good thing. (/) Done. I updated all occurrences of this. 6) Not an issue in your current code, but in case anyone comes in later to modify it I'd get rid of pointer aliases like the one below. You can still do the if(!whatever) on shared_ptrs, this is a case where constructing the shared_ptr with a pointer is a reasonable thing to do. (/) Fixed this for the FileSystem object. Additionally, I replaced SetOwnerState and SetPermissionState with just RecursionState to reduce boilerplate code, since they mostly match. A few function specific parameters are now passed though outside of this struct. > libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, > hdfs_chown, and hdfs_chmod > > > Key: HDFS-10754 > URL: https://issues.apache.org/jira/browse/HDFS-10754 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-10754.HDFS-8707.000.patch, > HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, > HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, > HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, > HDFS-10754.HDFS-8707.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod
[ https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anatoli Shein updated HDFS-10754: - Attachment: HDFS-10754.HDFS-8707.007.patch Please see the new patch. > libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, > hdfs_chown, and hdfs_chmod > > > Key: HDFS-10754 > URL: https://issues.apache.org/jira/browse/HDFS-10754 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein > Attachments: HDFS-10754.HDFS-8707.000.patch, > HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, > HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, > HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, > HDFS-10754.HDFS-8707.007.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10692) Point JDiff base version for HDFS from 2.6.0 to 2.7.2
[ https://issues.apache.org/jira/browse/HDFS-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431417#comment-15431417 ] Hudson commented on HDFS-10692: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10318 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10318/]) HDFS-10692. Update JDiff report's base version for HDFS from 2.6.0 to (vinodkv: rev dc7a1c54f4447fb01980ae61d5d67e90c4f52f00) * (add) hadoop-hdfs-project/hadoop-hdfs/dev-support/jdiff/Apache_Hadoop_HDFS_2.7.2.xml > Point JDiff base version for HDFS from 2.6.0 to 2.7.2 > - > > Key: HDFS-10692 > URL: https://issues.apache.org/jira/browse/HDFS-10692 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 3.0.0-alpha1-jdiff-hdfs.zip, HDFS-10692.1.patch, > HDFS-10692.2.patch > > > Now JDiff is pointed to 2.6.0, we need to upgrade it to the latest stable > release (2.7.2) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10692) Point JDiff base version for HDFS from 2.6.0 to 2.7.2
[ https://issues.apache.org/jira/browse/HDFS-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated HDFS-10692: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha1 2.8.0 Status: Resolved (was: Patch Available) Fixed the whitespace issue as part of the commit. Pushed this to trunk, branch-3.0.0-alpha1, branch-2 and branch-2.8. Thanks [~leftnoteasy]! > Point JDiff base version for HDFS from 2.6.0 to 2.7.2 > - > > Key: HDFS-10692 > URL: https://issues.apache.org/jira/browse/HDFS-10692 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 3.0.0-alpha1-jdiff-hdfs.zip, HDFS-10692.1.patch, > HDFS-10692.2.patch > > > Now JDiff is pointed to 2.6.0, we need to upgrade it to the latest stable > release (2.7.2) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-8986:
Attachment: HDFS-8986.07.patch
Patch 7 for pre-commit warnings.
> Add option to -du to calculate directory space usage excluding snapshots
>
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch,
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch,
> HDFS-8986.06.patch, HDFS-8986.07.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its
> children), the report includes space consumed by blocks that are only present
> in the snapshots. This is confusing for end users.
> {noformat}
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0 0 /tmp/parent
> 0 0 /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related
> disk usage in the output
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431209#comment-15431209
]
Hadoop QA commented on HDFS-8986:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 28s{color} | {color:orange} root: The patch generated 3 new + 352 unchanged
- 11 fixed = 355 total (was 363) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
9s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 56m
35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824869/HDFS-8986.06.patch |
| JIRA Issue | HDFS-8986 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml cc |
| uname | Linux 8c9103869a99 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-10719) In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.
[
https://issues.apache.org/jira/browse/HDFS-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431140#comment-15431140
]
Hadoop QA commented on HDFS-10719:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 27s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 3 new + 14 unchanged - 0 fixed = 17 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
54s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824853/HDFS-10719-3.patch |
| JIRA Issue | HDFS-10719 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 96163359e55b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16498/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16498/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16498/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> In HA, Namenode is failed to start If any of the Quorum hostname is
> unresolved.
> ---
>
> Key: HDFS-10719
> URL:
[jira] [Updated] (HDFS-10738) Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure
[
https://issues.apache.org/jira/browse/HDFS-10738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10738:
--
Fix Version/s: (was: 2.9.0)
2.8.0
> Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test
> failure
>
>
> Key: HDFS-10738
> URL: https://issues.apache.org/jira/browse/HDFS-10738
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Rakesh R
>Assignee: Rakesh R
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10738-00.patch, HDFS-10738-01.patch
>
>
> This jira is to analyse and fix the test case failure, which is failing in
> Jenkins build,
> [Build_16326|https://builds.apache.org/job/PreCommit-HDFS-Build/16326/testReport/org.apache.hadoop.security/TestRefreshUserMappings/testRefreshSuperUserGroupsConfiguration/]
> very frequently.
> {code}
> Error Message
> first auth for user2 should've succeeded: User: super_userL is not allowed to
> impersonate userL2
> Stacktrace
> java.lang.AssertionError: first auth for user2 should've succeeded: User:
> super_userL is not allowed to impersonate userL2
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.security.TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration(TestRefreshUserMappings.java:200)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10780) Block replication not happening on removing a volume when data being written to a datanode -- TestDataNodeHotSwapVolumes fails
[
https://issues.apache.org/jira/browse/HDFS-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430997#comment-15430997
]
Rushabh S Shah commented on HDFS-10780:
---
Is this somehow related to HDFS-9781 ?
> Block replication not happening on removing a volume when data being written
> to a datanode -- TestDataNodeHotSwapVolumes fails
> --
>
> Key: HDFS-10780
> URL: https://issues.apache.org/jira/browse/HDFS-10780
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>
> TestDataNodeHotSwapVolumes occasionally fails in the unit test
> testRemoveVolumeBeingWrittenForDatanode. Data write pipeline can have issues
> as there could be timeouts, data node not reachable etc, and in this test
> case it was more of induced one as one of the volumes in a datanode is
> removed while block write is in progress. Digging further in the logs, when
> the problem happens in the write pipeline, the error recovery is not
> happening as expected leading to block replication never catching up.
> Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 44.495 sec
> <<< FAILURE! - in org.apache.hadoop.hdfs.serv
> testRemoveVolumeBeingWritten(org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes)
> Time elapsed: 44.354 se
> java.util.concurrent.TimeoutException: Timed out waiting for /test to reach 3
> replicas
> Results :
> Tests in error:
>
> TestDataNodeHotSwapVolumes.testRemoveVolumeBeingWritten:637->testRemoveVolumeBeingWrittenForDatanode:714
> » Timeout
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> Following exceptions are not expected in this test run
> {noformat}
> 614 2016-08-10 12:30:11,269 [DataXceiver for client
> DFSClient_NONMAPREDUCE_-640082112_10 at /127.0.0.1:58805 [Receiving block
> BP-1852988604-172.16.3.66-1470857409044:blk_1073741825_1001]] DEBUG
> datanode.Da taNode (DataXceiver.java:run(320)) - 127.0.0.1:58789:Number
> of active connections is: 2
> 615 java.lang.IllegalMonitorStateException
> 616 at java.lang.Object.wait(Native Method)
> 617 at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList.waitVolumeRemoved(FsVolumeList.java:280)
> 618 at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.removeVolumes(FsDatasetImpl.java:517)
> 619 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.removeVolumes(DataNode.java:832)
> 620 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.removeVolumes(DataNode.java:798)
> {noformat}
> {noformat}
> 720 2016-08-10 12:30:11,287 [DataNode:
> [[[DISK]file:/Users/manoj/work/ups-hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data1/,
> [DISK]file:/Users/manoj/work/ups-hadoop/hadoop-hdfs-projec
> t/hadoop-hdfs/target/test/data/dfs/data/data2/]] heartbeating to
> localhost/127.0.0.1:58788] ERROR datanode.DataNode
> (BPServiceActor.java:run(768)) - Exception in BPOfferService for Block pool
> BP-18529 88604-172.16.3.66-1470857409044 (Datanode Uuid
> 711d58ad-919d-4350-af1e-99fa0b061244) service to localhost/127.0.0.1:58788
> 721 java.lang.NullPointerException
> 722 at
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1841)
> 723 at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:336)
> 724 at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:624)
> 725 at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:766)
> 726 at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots
[
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-8986:
--
Attachment: HDFS-8986.06.patch
Re-trigger the build.
+1 pending Jenkins.
> Add option to -du to calculate directory space usage excluding snapshots
>
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch,
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, HDFS-8986.06.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its
> children), the report includes space consumed by blocks that are only present
> in the snapshots. This is confusing for end users.
> {noformat}
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M 2.3 G /tmp/parent
> 799.7 M 2.3 G /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0 0 /tmp/parent
> 0 0 /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related
> disk usage in the output
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10719) In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.
[ https://issues.apache.org/jira/browse/HDFS-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Palanisamy updated HDFS-10719: -- Attachment: HDFS-10719-3.patch > In HA, Namenode is failed to start If any of the Quorum hostname is > unresolved. > --- > > Key: HDFS-10719 > URL: https://issues.apache.org/jira/browse/HDFS-10719 > Project: Hadoop HDFS > Issue Type: Bug > Components: journal-node, namenode >Affects Versions: 2.7.1 > Environment: HDP-2.4 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy > Labels: patch > Attachments: HDFS-10719-1.patch, HDFS-10719-2.patch, > HDFS-10719-3.patch > > > 2016-08-03 02:53:53,760 ERROR namenode.NameNode (NameNode.java:main(1712)) - > Failed to start namenode. > java.lang.IllegalArgumentException: Unable to construct journal, > qjournal://1:8485;2:8485;3:8485/shva > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1637) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:260) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:789) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:634) > at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:951) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:935) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at > org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1635) > ... 13 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107) > at > org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91) > at > org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.(IPCLoggerChannel.java:178) > at > org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156) > at > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367) > at > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149) > at > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.(QuorumJournalManager.java:116) > at > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.(QuorumJournalManager.java:105) > ... 18 more > 2016-08-03 02:53:53,765 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - > Exiting with status 1 > 2016-08-03 02:53:53,768 INFO namenode.NameNode (LogAdapter.java:info(47)) - > SHUTDOWN_MSG: > *and the failover is not successful* > I have attached the patch, It allows the Namenode to start if the majority of > the Quorums are resolvable. > throws warning if the quorum is unresolvable. > throws Unknown host exception if the majority of the journals are > unresolvable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods
[ https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430733#comment-15430733 ] Kihwal Lee commented on HDFS-10762: --- I will try finishing the review today. > Pass IIP for file status related methods > > > Key: HDFS-10762 > URL: https://issues.apache.org/jira/browse/HDFS-10762 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-10762.patch > > > The frequently called file status methods will not require path re-resolves > if the IIP is passed down the call stack. The code can be simplified further > if the IIP tracks if the original path was a reserved raw path. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430723#comment-15430723
]
Hadoop QA commented on HDFS-10783:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 92 unchanged - 0 fixed = 93 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824816/HDFS-10783.002.patch |
| JIRA Issue | HDFS-10783 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux da47faae517b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16496/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16496/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16496/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16496/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430699#comment-15430699
]
Hadoop QA commented on HDFS-10778:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m
0s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824818/HDFS-10778.004.patch |
| JIRA Issue | HDFS-10778 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 80f2cd6b9572 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16497/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16497/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16497/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
[jira] [Updated] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10778:
-
Attachment: HDFS-10778.004.patch
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch, HDFS-10778.004.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430587#comment-15430587
]
Yiqun Lin commented on HDFS-10778:
--
Attach a new patch to fix checkstyle warnings.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10783:
-
Attachment: HDFS-10783.002.patch
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -maxSize
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430578#comment-15430578
]
Yiqun Lin commented on HDFS-10783:
--
Thanks [~ajisakaa] for the review! Attach the patch for addressing your comment.
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -maxSize
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430352#comment-15430352
]
Hadoop QA commented on HDFS-10783:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 92 unchanged - 0 fixed = 93 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824787/HDFS-10783.001.patch |
| JIRA Issue | HDFS-10783 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 065bbfbd604d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16494/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16494/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16494/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16494/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> The option '-maxSize' and '-step' fail in OfflineImageViewer
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430348#comment-15430348
]
Hadoop QA commented on HDFS-10778:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestBlockStoragePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12824789/HDFS-10778.003.patch |
| JIRA Issue | HDFS-10778 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3453b4065648 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 115ecb5 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16495/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[
https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430316#comment-15430316
]
Kai Zheng commented on HDFS-8905:
-
Another comment is, ref. the following block, it's good to avoid using the
{{infoLock}} here, but I guess we don't need the {{synchronized}} keyword now
since {{ReadStatistics}} has already made its all methods protected.
{code}
- synchronized (infoLock) {
+ synchronized (readStatistics) {
readStatistics.addZeroCopyBytes(length);
}
{code}
> Refactor DFSInputStream#ReaderStrategy
> --
>
> Key: HDFS-8905
> URL: https://issues.apache.org/jira/browse/HDFS-8905
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Kai Zheng
>Assignee: SammiChen
> Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch,
> HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v13.patch,
> HDFS-8905-v2.patch, HDFS-8905-v3.patch, HDFS-8905-v4.patch,
> HDFS-8905-v5.patch, HDFS-8905-v6.patch, HDFS-8905-v7.patch,
> HDFS-8905-v8.patch, HDFS-8905-v9.patch
>
>
> DFSInputStream#ReaderStrategy family don't look very good. This refactors a
> little bit to make them make more sense.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430268#comment-15430268
]
Akira Ajisaka commented on HDFS-10783:
--
Thanks [~linyiqun] for the quick fix!
{code}
public void testOfflineImageViewerForFailedOptions() throws Throwable {
{code}
This method name looks ambiguous to me. I'm thinking it can be interpreted as
two different meanings
* Test for previously failed option (now succeeded)
* Test for options that have failed (now failing)
In this issue, "previously failed" is correct but it doesn't mean anything.
This is because generally regression tests are for fixing "previously failed"
something. Would you make the name more straightforward? (ex.
testOfflineImageViewerMaxSizeAndStepOptions)
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -maxSize
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430228#comment-15430228
]
Yiqun Lin commented on HDFS-10778:
--
Thanks [~ajisakaa] for the review.
{quote}
Would you file another jira for this issue and split the patch?
{quote}
Done. I created the HDFS-10783 for tracking this and posted a simple patch for
that.
Update the patch of this jira.
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10778) Optimize the output result of FileDistribution processor in hdfs oiv command
[
https://issues.apache.org/jira/browse/HDFS-10778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10778:
-
Attachment: HDFS-10778.003.patch
> Optimize the output result of FileDistribution processor in hdfs oiv command
>
>
> Key: HDFS-10778
> URL: https://issues.apache.org/jira/browse/HDFS-10778
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.7.1
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-10778.001.patch, HDFS-10778.002.patch,
> HDFS-10778.003.patch
>
>
> Now It's not directly to understand the output result of the
> {{FileDistribution}} processor that in hdfs oiv command for users. For
> example, this is a original output:
> {code}
> SizeNumFiles
> 0 22556
> 1048576 404971
> 2097152 29259
> 3145728 16937
> 4194304 9197
> 5242880 6889
> 6291456 4930
> 7340032 4070
> 8388608 299384
> 9437184 274623
> {code}
> Two aspects make that hard to understand for users.
> First, the size column just showed as the number in byte, it's not readable
> here. The better way is showed with a binary prefix.
> Second, the size column would be better to showed as a size range. It will
> let users know the value in {{NumFiles}} column was counted from A size to B
> size.
> The expected output result should be this:
> {code}
> Size Range NumFiles
> (0 B, 0 B] 1666332
> (0 B, 1 M]778473
> (1 M, 2 M] 35125
> (2 M, 3 M] 13978
> (3 M, 4 M] 10158
> (4 M, 5 M] 6970
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10783:
-
Attachment: HDFS-10783.001.patch
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -maxSize
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
[
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiqun Lin updated HDFS-10783:
-
Status: Patch Available (was: Open)
Attach a simple patch for this.
> The option '-maxSize' and '-step' fail in OfflineImageViewer
>
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -maxSize
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer
Yiqun Lin created HDFS-10783:
Summary: The option '-maxSize' and '-step' fail in
OfflineImageViewer
Key: HDFS-10783
URL: https://issues.apache.org/jira/browse/HDFS-10783
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the
following exception:
{code}
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
-maxSize
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6532) Intermittent test failure org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
[
https://issues.apache.org/jira/browse/HDFS-6532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-6532:
Attachment: TEST-org.apache.hadoop.hdfs.TestCrcCorruption.xml
I further looked at the issue but not root caused yet.
Seems to me the biggest difference between success and failure cases is that,
success cases have the following line, and then the retry terminates quickly.
No such line in the failure cases, then as Yiqun said, test timeout will cause
an interrupt exception thrown from {{close}} after 50 seconds.
{noformat}
2016-08-20 08:35:03,052 INFO datanode.DataNode
(DataXceiver.java:writeBlock(835)) - opWriteBlock
BP-1047920701-192.168.1.79-1471707299069:blk_1073741826_1008 received exception
java.io.InterruptedIOException: Interruped while waiting for IO on channel
java.nio.channels.SocketChannel[connected local=/127.0.0.1:51075
remote=/127.0.0.1:5]. 6 millis timeout left.
{noformat}
Attaching a full failure log for reference, since this has not failed locally
for me at all
I think there may be some race / corner cases causing said thread(s) not
interrupted when checksum error happens, but haven't found any clue about it so
far. :(
> Intermittent test failure
> org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
> --
>
> Key: HDFS-6532
> URL: https://issues.apache.org/jira/browse/HDFS-6532
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode, hdfs-client
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
> Attachments: TEST-org.apache.hadoop.hdfs.TestCrcCorruption.xml
>
>
> Per https://builds.apache.org/job/Hadoop-Hdfs-trunk/1774/testReport, we had
> the following failure. Local rerun is successful
> {code}
> Regression
> org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt
> Failing for the past 1 build (Since Failed#1774 )
> Took 50 sec.
> Error Message
> test timed out after 5 milliseconds
> Stacktrace
> java.lang.Exception: test timed out after 5 milliseconds
> at java.lang.Object.wait(Native Method)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:2024)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.flushInternal(DFSOutputStream.java:2008)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2107)
> at
> org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:70)
> at
> org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:98)
> at
> org.apache.hadoop.hdfs.TestCrcCorruption.testCorruptionDuringWrt(TestCrcCorruption.java:133)
> {code}
> See relevant exceptions in log
> {code}
> 2014-06-14 11:56:15,283 WARN datanode.DataNode
> (BlockReceiver.java:verifyChunks(404)) - Checksum error in block
> BP-1675558312-67.195.138.30-1402746971712:blk_1073741825_1001 from
> /127.0.0.1:41708
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> DFSClient_NONMAPREDUCE_-1139495951_8 at 64512 exp: 1379611785 got: -12163112
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:353)
> at
> org.apache.hadoop.util.DataChecksum.verifyChunkedSums(DataChecksum.java:284)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.verifyChunks(BlockReceiver.java:402)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:537)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:734)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
> at java.lang.Thread.run(Thread.java:662)
> 2014-06-14 11:56:15,285 WARN datanode.DataNode
> (BlockReceiver.java:run(1207)) - IOException in BlockReceiver.run():
> java.io.IOException: Shutting down writer and responder due to a checksum
> error in received data. The error response has been sent upstream.
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1352)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1278)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1199)
> at java.lang.Thread.run(Thread.java:662)
> ...
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8905) Refactor DFSInputStream#ReaderStrategy
[
https://issues.apache.org/jira/browse/HDFS-8905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15430113#comment-15430113
]
Kai Zheng commented on HDFS-8905:
-
Hi Sammi,
The latest change looks good. One comment for the {{readFromBuffer}} call,
would you help check if we need to forward the position of the {{src}} buffer?
Can we add some comment about this to note the position of the src buffer is
changed or not after the call? Thanks.
> Refactor DFSInputStream#ReaderStrategy
> --
>
> Key: HDFS-8905
> URL: https://issues.apache.org/jira/browse/HDFS-8905
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Reporter: Kai Zheng
>Assignee: SammiChen
> Attachments: HDFS-8905-HDFS-7285-v1.patch, HDFS-8905-v10.patch,
> HDFS-8905-v11.patch, HDFS-8905-v12.patch, HDFS-8905-v13.patch,
> HDFS-8905-v2.patch, HDFS-8905-v3.patch, HDFS-8905-v4.patch,
> HDFS-8905-v5.patch, HDFS-8905-v6.patch, HDFS-8905-v7.patch,
> HDFS-8905-v8.patch, HDFS-8905-v9.patch
>
>
> DFSInputStream#ReaderStrategy family don't look very good. This refactors a
> little bit to make them make more sense.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
