[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434210#comment-15434210
 ] 

Xiao Chen commented on HDFS-8986:
-

Thanks [~jojochuang] for triggering a new run!

- Checkstyle is same as trunk, more readable this way.
- whitespace is false negative, not related.
- All failed tests look unrelated, and passed locally (with jdk7).

> Add option to -du to calculate directory space usage excluding snapshots
> 
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch, 
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, 
> HDFS-8986.06.patch, HDFS-8986.07.patch, HDFS-8986.branch-2.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its 
> children), the report includes space consumed by blocks that are only present 
> in the snapshots. This is confusing for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related 
> disk usage in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10340) data node sudden killed

[KWON BYUNGCHANG (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434106#comment-15434106
 ] 

KWON BYUNGCHANG commented on HDFS-10340:


Yes. my cluster was configured same user of DataNode and Nodemanager due to my 
infrastructure have some limitation.
anyway I have figured it out using YARN-4459.

[~kihwal] Thank you for your advice.

> data node sudden killed 
> 
>
> Key: HDFS-10340
> URL: https://issues.apache.org/jira/browse/HDFS-10340
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0
> Environment: Ubuntu 16.04 LTS , RAM 16g , cpu core : 8 , hdd 100gb, 
> hadoop 2.6.0
>Reporter: tu nguyen khac
>Priority: Critical
>
> I tried to setup a new data node using ubuntu 16 
> and get it join to an existed Hadoop Hdfs cluster ( there are 10 nodes in 
> this cluster and they all run on centos Os 6 ) 
> But when i try to boostrap this node , after about 10 or 20 minutes i get 
> this strange errors : 
> 2016-04-26 20:12:09,394 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.3.24.65:55323, dest: /10.3.24.197:50010, bytes: 79902, op: HDFS_WRITE, 
> cliID: DFSClient_NONMAPREDUCE_1379996362_1, offset: 0, srvID: 
> 225f5b43-1dd3-4ac6-88d2-1e8d27dba55b, blockid: 
> BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, duration: 
> 15331628
> 2016-04-26 20:12:09,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> PacketResponder: BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, 
> type=LAST_IN_PIPELINE, downstreams=0:[] terminating
> 2016-04-26 20:12:25,410 INFO 
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification 
> succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038502_789829
> 2016-04-26 20:12:25,411 INFO 
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification 
> succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832
> 2016-04-26 20:13:18,546 INFO 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
>  Scheduling blk_1074038502_789829 file 
> /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502
>  for deletion
> 2016-04-26 20:13:18,562 INFO 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
>  Deleted BP-352432948-10.3.24.65-1433821675295 blk_1074038502_789829 file 
> /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502
> 2016-04-26 20:15:46,481 ERROR 
> org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
> 2016-04-26 20:15:46,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down DataNode at bigdata-dw-24-197/10.3.24.197
> /



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434097#comment-15434097
 ] 

Hadoop QA commented on HDFS-10742:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 109 unchanged - 0 fixed = 110 total (was 109) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
45s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 75m 
23s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be 
atomic in org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:may not be atomic in 
org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:[line 163] |
|  |  Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be 
atomic in org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:may not be atomic in 
org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:[line 179] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825170/HDFS-10742.004.patch |
| JIRA Issue | HDFS-10742 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 76b0721701a5 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / c37346d |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434094#comment-15434094
 ] 

Xiaoyu Yao commented on HDFS-10690:
---

I mean the LinkedMap patch should just work without revision. 
LinkedMap#get(Object obj) and LinkedMap#remove(Object obj) will work with key 
(Object type) instead of index (int type) because it is inherited from 
AbstractHashedMap as I mentioned in 
[comment.|https://issues.apache.org/jira/browse/HDFS-10690?focusedCommentId=15433269=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15433269].

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434085#comment-15434085
 ] 

Hadoop QA commented on HDFS-10742:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 2 new + 109 unchanged - 0 fixed = 111 total (was 109) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
54s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 29s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 23s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
|  |  Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be 
atomic in org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:may not be atomic in 
org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:[line 164] |
|  |  Sequence of calls to java.util.concurrent.ConcurrentHashMap may not be 
atomic in org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:may not be atomic in 
org.apache.hadoop.hdfs.InstrumentedLock.check(long, long)  At 
InstrumentedLock.java:[line 180] |
| Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825168/HDFS-10742.004.patch |
| JIRA Issue | HDFS-10742 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 29af66f6d836 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / c37346d |
| Default Java | 1.8.0_101 |
| findbugs | 

[jira] [Updated] (HDFS-9781) FsDatasetImpl#getBlockReports can occasionally throw NullPointerException

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-9781:

Assignee: Manoj Govindassamy  (was: Xiao Chen)

> FsDatasetImpl#getBlockReports can occasionally throw NullPointerException
> -
>
> Key: HDFS-9781
> URL: https://issues.apache.org/jira/browse/HDFS-9781
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 3.0.0-alpha1
> Environment: Jenkins
>Reporter: Wei-Chiu Chuang
>Assignee: Manoj Govindassamy
> Attachments: HDFS-9781.01.patch
>
>
> FsDatasetImpl#getBlockReports occasionally throws NPE. The NPE caused 
> TestFsDatasetImpl#testRemoveVolumeBeingWritten to time out, because the test 
> waits for the call to FsDatasetImpl#getBlockReports to complete without 
> exceptions.
> Additionally, the test should be updated to identify an expected exception, 
> using {{GenericTestUtils.assertExceptionContains()}}
> {noformat}
> Exception in thread "Thread-20" java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.getBlockReports(FsDatasetImpl.java:1709)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1BlockReportThread.run(TestFsDatasetImpl.java:587)
> 2016-02-08 15:47:30,379 [Thread-21] WARN  impl.TestFsDatasetImpl 
> (TestFsDatasetImpl.java:run(606)) - Exception caught. This should not affect 
> the test
> java.io.IOException: Failed to move meta file for ReplicaBeingWritten, 
> blk_0_0, RBW
>   getNumBytes() = 0
>   getBytesOnDisk()  = 0
>   getVisibleLength()= 0
>   getVolume()   = 
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current
>   getBlockFile()= 
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0
>   bytesAcked=0
>   bytesOnDisk=0 from 
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta
>  to 
> /home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:857)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.addFinalizedBlock(BlockPoolSlice.java:295)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addFinalizedBlock(FsVolumeImpl.java:819)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeReplica(FsDatasetImpl.java:1620)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.finalizeBlock(FsDatasetImpl.java:1601)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl$1ResponderThread.run(TestFsDatasetImpl.java:603)
> Caused by: java.io.IOException: 
> renameTo(src=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/rbw/blk_0_0.meta,
>  
> dst=/home/weichiu/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/Nmi6rYndvr/data0/current/bpid-0/current/finalized/subdir0/subdir0/blk_0_0.meta)
>  failed.
> at org.apache.hadoop.io.nativeio.NativeIO.renameTo(NativeIO.java:873)
> at 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.moveBlockFiles(FsDatasetImpl.java:855)
> ... 5 more
> 2016-02-08 15:47:34,381 [Thread-19] INFO  impl.FsDatasetImpl 
> (FsVolumeList.java:waitVolumeRemoved(287)) - Volume reference is released.
> 2016-02-08 15:47:34,384 [Thread-19] INFO  impl.TestFsDatasetImpl 
> (TestFsDatasetImpl.java:testRemoveVolumeBeingWritten(622)) - Volumes removed
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Arpit Agarwal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434078#comment-15434078
 ] 

Arpit Agarwal commented on HDFS-10742:
--

bq. On a related note, the virtues of AutoCloseableLock (HADOOP-13466) are not 
obvious to me. Static analysis tools already catch what it enforces with 
AutoCloseable. Its interface is almost identical to 
java.util.concurrent.locks.Lock (admitting extensions like this). It seems to 
add an idiosyncrasy where there is already an idiom (with supporting tooling). 
Is there another advantage?
Hi Chris, there is no other advantage. It just eliminates the {{finally}} block.

[~vagarychen], I'll take a look at your patch.

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch, HDFS-10742.004.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434077#comment-15434077
 ] 

Fenghua Hu commented on HDFS-10690:
---

I got you. I'll run YCSB.

Regarding the patch, what do you mean "I don't think we need the extra 
indexOf()."? I think the LinkedMap patch needs revision, am i right?

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10706) Add tool generating FSImage from external store

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434047#comment-15434047
 ] 

Hadoop QA commented on HDFS-10706:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  4m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
 5s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
40s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
25s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
36s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  2m 
43s{color} | {color:green} HDFS-9806 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
22s{color} | {color:green} HDFS-9806 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
12s{color} | {color:green} HDFS-9806 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
10s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 31s{color} | {color:orange} root: The patch generated 45 new + 6 unchanged - 
0 fixed = 51 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-tools {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 44s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
51s{color} | {color:green} hadoop-fs2img in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 36m 
56s{color} | {color:green} hadoop-tools in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 58s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
|   | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825146/HDFS-10706-HDFS-9806.002.patch
 |
| JIRA 

[jira] [Comment Edited] (HDFS-10780) Block replication not happening on removing a volume when data being written to a datanode -- TestDataNodeHotSwapVolumes fails

[Manoj Govindassamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433995#comment-15433995
 ] 

Manoj Govindassamy edited comment on HDFS-10780 at 8/24/16 1:14 AM:


*Summary:* 
-- I can see an issue in the current code where by the NameNode can miss out 
permantnetly to replicate a block to a DataNode. 
-- If the write pipeline doesn’t have enough DataNodes to match the expected 
replication factor, then NameNode schedules replication of the block moments 
after the block has been COMMITTED or COMPLETED.  But, if the block COMMIT at 
NameNode arrives/happens after all BlockManager::addStoredBlock() (happens when 
processing block reports from DataNodes in the current write pipeline) then 
NameNode totally misses out to replicate the block because of the way it 
manages the {{LowRedundancyBlocks}} and {{PendingReconstructionBlocks}}.

*Details:*
Here are the events and their rough timeline order I noticed in one of the 
failures.
1. Say the setup has 3 DataNodes ( DN(1..3) ) and the replication factor 
configured to be 3
2. Client is writing a block (BLK_xyz_GS1) with the constructed pipeline DN1 —> 
DN2 —> DN3
3. Say DN1 encounters an issue (like downstream communication, or interrupts or 
storage volume issue etc.,) before the block is FINALIZED
4. Client detects the pipeline issue and gets a new write pipeline DN2 —> DN3 
5. Generation stamp for the current block is bumped up (BLK_xyz_GS2)
6. Client write (BLK_xyz_GS2) to DN2 and DN3
7. NameNode (NN) is getting Incremental Block Reports (IBR) from DN1, DN2 and  
DN3
8. NN sees a mix of (BLK_xyz_GS1) and (BLK_xyz_GS2) in IBRs from DN1, DN2 and 
DN3
9. NN marks (BLK_xyz_GS1) as corrupted and puts these DNs in invalidation list 
to remove that particular block

10. After seeing the first BLK_xyz_GS2 from one the DNs (say DN2)
— — NN adds  BLK_xyz_GS2 to the respective {{DatanodeStorageInfo}} (Refer: 
{{BlockManager::addStoredBlock}} )
— — Since  BLK_xyz_GS2 is not COMMITTED from the Client yet, NN cannot move 
BLK_xyz_GS2 to COMPLETE state
— — NN does not update the {{LowRedundancyBlocks}} neededReconstruction as it 
is done only after BLK_xyz_GS2 is COMMITTED by the Client 
11. NN sees BLK_xyz_GS2 from the other DN (DN3)
— — NN adds BLK_xyz_GS2
— — At NN, BLK_xyz_GS2 is still in UNDER_CONSTRUCTION state as the Client has 
not COMMITTED yet. So, neededReconstruction is still not updated.


12. At this point, for BLK_xyz_GS2 the live replica count is 2 and the expected 
replica count is 3.
— — For the {{ReplicationMonitor}} to pick up a replica work, the intention 
must be available in {{LowRedundancyBlocks}} neededReconstruction
— — Since no event triggered the addition of intention yet, no block 
replication work scheduled yet to the missing DN1

13. Client closes the file
— — NN moves the block to COMMITTED state
— — Since there are already 2 love copies of the block, NN moves the block to 
COMPLETED state
— — But, before moving the block to COMPLETED state, NN does something like 
below 

{{BlockManager.java}}
{noformat}
public boolean commitOrCompleteLastBlock(BlockCollection bc,
Block 
commitBlock) throws IOException {

  ...
  final boolean committed = commitBlock(lastBlock, commitBlock);
  
  if (committed && lastBlock.isStriped()) {
  
  ...
  if (hasMinStorage(lastBlock)) {

if (committed) {
  
  //XXX: Is adding to {{PendingReconstructionBlocks}} list without 
adding to
  // {{LowRedundancyBlocks}} list right ?
  // This fakes block replication task in progress without even
  // any {{BlockReconstructionWork}} scheduled. 

  addExpectedReplicasToPending(lastBlock);

 }

completeBlock(lastBlock, false);

 }
  
... 

}
{noformat}

— — Since the block is added the {{pendingReconstruction}} list without any 
actual reconstruction work, the follow on {{checkRedundancy}} mistakenly 
believes that there is sufficient redundancy for the block and does not trigger 
any further block reconstruction works.

{noformat}
839 2016-08-23 10:57:21,450 [IPC Server handler 4 on 57980] INFO  
blockmanagement.BlockManager (BlockManager.java:checkRedundancy(4046)) - 
CheckRedun Block: blk_1073741825_1002, exp: 3, replica: *LIVE=2*, READONLY=0, 
DECOMMISSIONING=0, DECOMMISSIONED=0, CORRUPT=0, EXCESS=0, STALESTORAGE= 0, 
REDUNDANT=0, *pending: 1*   
840 2016-08-23 10:57:21,451 [IPC Server handler 4 on 57980] INFO  
blockmanagement.BlockManager 
(BlockManager.java:hasEnoughEffectiveReplicas(1685)) - Blk: 
blk_1073741825_1002, numEffectiveReplicas: 3, required: 3, pendingReplicaNum: 
1, placementPolicy: true
{noformat}

— — {{ReplicationMonitor}} which runs once in every 3 seconds, is also unable 
to help here as {{LowRedundancyBlocks}} list {{neededReplication}} is empty. 

— — So the test fails with following signature 

{noformat}

[jira] [Commented] (HDFS-10780) Block replication not happening on removing a volume when data being written to a datanode -- TestDataNodeHotSwapVolumes fails

[Manoj Govindassamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433995#comment-15433995
 ] 

Manoj Govindassamy commented on HDFS-10780:
---


*Summary:* 
-- I can see an issue in the current code where by the NameNode can miss out 
permantnetly to replicate a block to a DataNode. 
-- If the write pipeline doesn’t have enough DataNodes to match the expected 
replication factor, then NameNode schedules replication of the block moments 
after the block has been COMMITTED or COMPLETED.  But, if the block COMMIT at 
NameNode arrives/happens after all BlockManager::addStoredBlock() (happens when 
processing block reports from DataNodes in the current write pipeline) then 
NameNode totally misses out to replicate the block because of the way it 
manages the {[LowRedundancyBlocks}} and {{PendingReconstructionBlocks}}.

*Details:*
Here are the events and their rough timeline order I noticed in one of the 
failures.
1. Say the setup has 3 DataNodes ( DN(1..3) ) and the replication factor 
configured to be 3
2. Client is writing a block (BLK_xyz_GS1) with the constructed pipeline DN1 —> 
DN2 —> DN3
3. Say DN1 encounters an issue (like downstream communication, or interrupts or 
storage volume issue etc.,) before the block is FINALIZED
4. Client detects the pipeline issue and gets a new write pipeline DN2 —> DN3 
5. Generation stamp for the current block is bumped up (BLK_xyz_GS2)
6. Client write (BLK_xyz_GS2) to DN2 and DN3
7. NameNode (NN) is getting Incremental Block Reports (IBR) from DN1, DN2 and  
DN3
8. NN sees a mix of (BLK_xyz_GS1) and (BLK_xyz_GS2) in IBRs from DN1, DN2 and 
DN3
9. NN marks (BLK_xyz_GS1) as corrupted and puts these DNs in invalidation list 
to remove that particular block

10. After seeing the first BLK_xyz_GS2 from one the DNs (say DN2)
— — NN adds  BLK_xyz_GS2 to the respective {{DatanodeStorageInfo}} (Refer: 
{{BlockManager::addStoredBlock}} )
— — Since  BLK_xyz_GS2 is not COMMITTED from the Client yet, NN cannot move 
BLK_xyz_GS2 to COMPLETE state
— — NN does not update the {{LowRedundancyBlocks}} neededReconstruction as it 
is done only after BLK_xyz_GS2 is COMMITTED by the Client 
11. NN sees BLK_xyz_GS2 from the other DN (DN3)
— — NN adds BLK_xyz_GS2
— — At NN, BLK_xyz_GS2 is still in UNDER_CONSTRUCTION state as the Client has 
not COMMITTED yet. So, neededReconstruction is still not updated.


12. At this point, for BLK_xyz_GS2 the live replica count is 2 and the expected 
replica count is 3.
— — For the {{ReplicationMonitor}} to pick up a replica work, the intention 
must be available in {{LowRedundancyBlocks}} neededReconstruction
— — Since no event triggered the addition of intention yet, no block 
replication work scheduled yet to the missing DN1

13. Client closes the file
— — NN moves the block to COMMITTED state
— — Since there are already 2 love copies of the block, NN moves the block to 
COMPLETED state
— — But, before moving the block to COMPLETED state, NN does something like 
below 

{{BlockManager.java}}
{noformat}
public boolean commitOrCompleteLastBlock(BlockCollection bc,
Block 
commitBlock) throws IOException {
  ...
  final boolean committed = commitBlock(lastBlock, commitBlock);
  if 
(committed && lastBlock.isStriped()) {
  ...
  if (hasMinStorage(lastBlock)) {
if (committed) {
  //XXX: Is adding 
to {{PendingReconstructionBlocks}} list without adding to 
  // {{LowRedundancyBlocks}} list right ?
  // This fakes block replication task in progress without even
  // any {{BlockReconstructionWork}} scheduled.
  
addExpectedReplicasToPending(lastBlock);
}
completeBlock(lastBlock, 
false);
  }
  ... 
}

{noformat}

— — Since the block is added the {{pendingReconstruction}} list without any 
actual reconstruction work, the follow on {{checkRedundancy}} mistakenly 
believes that there is sufficient redundancy for the block and does not trigger 
any further block reconstruction works.

{noformat}
 839 2016-08-23 10:57:21,450 [IPC Server handler 4 on 57980] INFO  
blockmanagement.BlockManager (BlockManager.java:checkRedundancy(4046)) - 
CheckRedun Block: blk_1073741825_1002, exp: 3, replica: *LIVE=2*, READONLY=0, 
DECOMMISSIONING=0, DECOMMISSIONED=0, CORRUPT=0, EXCESS=0, STALESTORAGE= 0, 
REDUNDANT=0, *pending: 1*

 840 2016-08-23 10:57:21,451 [IPC Server handler 4 on 57980] INFO  
blockmanagement.BlockManager 
(BlockManager.java:hasEnoughEffectiveReplicas(1685)) - Blk: 
blk_1073741825_1002, numEffectiveReplicas: 3, required: 3, pendingReplicaNum: 
1, placementPolicy: true

{noformat}

— — {{ReplicationMonitor}} which runs once in every 3 seconds, is also unable 
to help here as {{LowRedundancyBlocks}} list {{neededReplication}} is empty. 

— — So the test fails with following signature 

{{noformat}}
Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
Tests 

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433988#comment-15433988
 ] 

Xiaoyu Yao commented on HDFS-10690:
---

bq.  I'll write a micro benchmark to compare LinkedMap with TreeMap and get 
back to you.

We have a lot of variations here: key, hash function, percentage of 
insert/delete/lookup operations, etc. There are quite some micro benchmark for 
TreeMap/LinkedHashMap(similar to LinkedMap from apache common) if you search 
online. Instead of tuning the parameter to simulate the workloads with micro 
benchmark, I would suggest we run the same YCSB benchmark with the last patch 
using LinkedMap for easy comparison. 

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Chen Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433987#comment-15433987
 ] 

Chen Liang commented on HDFS-10742:
---

Thanks for the catch [~chris.douglas]! Updated a patch that addresses most of 
the issues. (Changing to slf4j logger is a bit tricky as I'm directly using the 
log from FsDatasetImpl here). 

Regarding you note, the intentions of {{AutoCloseable}} was mainly to allow 
easy instrumentation such as the instrumented lock in this JIRA. I'm actually 
not quit familiar with the static analysis tools as you mentioned. Anything to 
add [~arpitagarwal]?

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch, HDFS-10742.004.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-10742:
--
Attachment: HDFS-10742.004.patch

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch, HDFS-10742.004.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-10742:
--
Attachment: (was: HDFS-10742.004.patch)

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch, HDFS-10742.004.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Chen Liang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chen Liang updated HDFS-10742:
--
Attachment: HDFS-10742.004.patch

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch, HDFS-10742.004.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer

[Yiqun Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433957#comment-15433957
 ] 

Yiqun Lin commented on HDFS-10783:
--

Thanks a lot for the commit!

> The option '-maxSize' and '-step' fail in OfflineImageViewer
> 
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch, 
> HDFS-10783.003.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the 
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: 
> -maxSize
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10788) fsck NullPointerException when it encounters corrupt replicas

[Jeff Field (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433950#comment-15433950
 ] 

Jeff Field edited comment on HDFS-10788 at 8/24/16 12:40 AM:
-

For anyone that does encounter the problem, once you fsck your way down to a 
list of problem directories, you can generate a list of bad files themselves 
with:

{code}
for i in `cat baddirs.txt`; do HADOOP_ROOT_LOGGER=TRACE,console hdfs dfs 
-checksum $i/\* 2>&1 > /dev/null | grep -v DEBUG | grep -B1 
"org.apache.hadoop.ipc.RemoteException" | head -n1 | cut -d\" -f2 >> 
badfiles.txt; done
{code}


was (Author: jfield):
For anyone that does encounter the problem, once you fsck your way down to a 
list of problem directories, you can generate a list of bad files themselves 
with:

{code}
for i in `cat bardirs.txt`; do HADOOP_ROOT_LOGGER=TRACE,console hdfs dfs 
-checksum $i/\* 2>&1 > /dev/null | grep -v DEBUG | grep -B1 
"org.apache.hadoop.ipc.RemoteException" | head -n1 | cut -d\" -f2 >> 
badfiles.txt; done
{code}

> fsck NullPointerException when it encounters corrupt replicas
> -
>
> Key: HDFS-10788
> URL: https://issues.apache.org/jira/browse/HDFS-10788
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.0
> Environment: CDH5.5.2, CentOS 6.7
>Reporter: Jeff Field
>
> Somehow (I haven't found root cause yet) we ended up with blocks that have 
> corrupt replicas where the replica count is inconsistent between the blockmap 
> and the corrupt replicas map. If we try to hdfs fsck any parent directory 
> that has a child with one of these blocks, fsck will exit with something like 
> this:
> {code}
> $ hdfs fsck /path/to/parent/dir/ | egrep -v '^\.+$'
> Connecting to namenode via http://mynamenode:50070
> FSCK started by bot-hadoop (auth:KERBEROS_SSL) from /10.97.132.43 for path 
> /path/to/parent/dir/ at Tue Aug 23 20:34:58 UTC 2016
> .FSCK 
> ended at Tue Aug 23 20:34:59 UTC 2016 in 1098 milliseconds
> null
> Fsck on path '/path/to/parent/dir/' FAILED
> {code}
> So I start at the top, fscking every subdirectory until I find one or more 
> that fails. Then I do the same thing with those directories (our top level 
> directories all have subdirectories with date directories in them, which then 
> contain the files) and once I find a directory with files in it, I run a 
> checksum of the files in that directory. When I do that, I don't get the name 
> of the file, rather I get:
> checksum: java.lang.NullPointerException
> but since the files are in order, I can figure it out by seeing which file 
> was before the NPE. Once I get to this point, I can see the following in the 
> namenode log when I try to checksum the corrupt file:
> 2016-08-23 20:24:59,627 WARN 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent 
> number of corrupt replicas for blk_1335893388_1100036319546 blockMap has 0 
> but corrupt replicas map has 1
> 2016-08-23 20:24:59,627 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 23 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 192.168.1.100:47785 Call#1 Retry#0
> java.lang.NullPointerException
> At which point I can delete the file, but it is a very tedious process.
> Ideally, shouldn't fsck be able to emit the name of the file that is the 
> source of the problem - and (if -delete is specified) get rid of the file, 
> instead of exiting without saying why?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10788) fsck NullPointerException when it encounters corrupt replicas

[Jeff Field (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433950#comment-15433950
 ] 

Jeff Field commented on HDFS-10788:
---

For anyone that does encounter the problem, once you fsck your way down to a 
list of problem directories, you can generate a list of bad files themselves 
with:

{code}
for i in `cat bardirs.txt`; do HADOOP_ROOT_LOGGER=TRACE,console hdfs dfs 
-checksum $i/\* 2>&1 > /dev/null | grep -v DEBUG | grep -B1 
"org.apache.hadoop.ipc.RemoteException" | head -n1 | cut -d\" -f2 >> 
badfiles.txt; done
{code}

> fsck NullPointerException when it encounters corrupt replicas
> -
>
> Key: HDFS-10788
> URL: https://issues.apache.org/jira/browse/HDFS-10788
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.0
> Environment: CDH5.5.2, CentOS 6.7
>Reporter: Jeff Field
>
> Somehow (I haven't found root cause yet) we ended up with blocks that have 
> corrupt replicas where the replica count is inconsistent between the blockmap 
> and the corrupt replicas map. If we try to hdfs fsck any parent directory 
> that has a child with one of these blocks, fsck will exit with something like 
> this:
> {code}
> $ hdfs fsck /path/to/parent/dir/ | egrep -v '^\.+$'
> Connecting to namenode via http://mynamenode:50070
> FSCK started by bot-hadoop (auth:KERBEROS_SSL) from /10.97.132.43 for path 
> /path/to/parent/dir/ at Tue Aug 23 20:34:58 UTC 2016
> .FSCK 
> ended at Tue Aug 23 20:34:59 UTC 2016 in 1098 milliseconds
> null
> Fsck on path '/path/to/parent/dir/' FAILED
> {code}
> So I start at the top, fscking every subdirectory until I find one or more 
> that fails. Then I do the same thing with those directories (our top level 
> directories all have subdirectories with date directories in them, which then 
> contain the files) and once I find a directory with files in it, I run a 
> checksum of the files in that directory. When I do that, I don't get the name 
> of the file, rather I get:
> checksum: java.lang.NullPointerException
> but since the files are in order, I can figure it out by seeing which file 
> was before the NPE. Once I get to this point, I can see the following in the 
> namenode log when I try to checksum the corrupt file:
> 2016-08-23 20:24:59,627 WARN 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent 
> number of corrupt replicas for blk_1335893388_1100036319546 blockMap has 0 
> but corrupt replicas map has 1
> 2016-08-23 20:24:59,627 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 23 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 192.168.1.100:47785 Call#1 Retry#0
> java.lang.NullPointerException
> At which point I can delete the file, but it is a very tedious process.
> Ideally, shouldn't fsck be able to emit the name of the file that is the 
> source of the problem - and (if -delete is specified) get rid of the file, 
> instead of exiting without saying why?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10756) Expose getTrashRoot to HTTPFS and WebHDFS

[Yuanbo Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433922#comment-15433922
 ] 

Yuanbo Liu commented on HDFS-10756:
---

[~jojochuang] and [~xiaochen] Thanks a lot for your comments.
I will read through your suggestions and response them later.
Thanks again for your time!

> Expose getTrashRoot to HTTPFS and WebHDFS
> -
>
> Key: HDFS-10756
> URL: https://issues.apache.org/jira/browse/HDFS-10756
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: encryption, httpfs, webhdfs
>Reporter: Xiao Chen
>Assignee: Yuanbo Liu
> Attachments: HDFS-10756.001.patch
>
>
> Currently, hadoop FileSystem API has 
> [getTrashRoot|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2708]
>  to determine trash directory at run time. Default trash dir is under 
> {{/user/$USER}}
> For an encrypted file, since moving files between/in/out of EZs are not 
> allowed, when an EZ file is deleted via CLI, it calls in to [DFS 
> implementation|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2485]
>  to move the file to a trash directory under the same EZ.
> This works perfectly fine for CLI users or java users who call FileSystem 
> API. But for users via httpfs/webhdfs, currently there is no way to figure 
> out what the trash root would be. This jira is proposing we add such 
> interface to httpfs and webhdfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433918#comment-15433918
 ] 

Fenghua Hu commented on HDFS-10690:
---

[~xyao], thanks for your comments. I'll write a micro benchmark to compare 
LinkedMap with TreeMap and get back to you.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.

[Joe Pallas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433904#comment-15433904
 ] 

Joe Pallas commented on HDFS-10636:
---

Possibly the only requirement to subclass {{FinalizedReplica}} today is that it 
appears in the dataset SPI, so any alternative implementation has to use it.  
The proposal removes that, so maybe there isn't a strong reason to have it be a 
type (if I understand the distinction I think [~virajith] is making between 
class as type and class as state).  There are several places in the current 
patch where {{FinalizedReplica}} becomes {{ReplicaInfo}}, and I think that's a 
reasonable change.  

The concern [~eddyxu] expressed above about states and types is valid, but, in 
this case, I think the type system isn't really helping the programmer very 
much.  Existing code using {{FinalizedReplica}} is already frequently stuck 
with only {{ReplicaInfo}} known at compile time, so a state check is needed 
anyway, and using a cast to {{FinalizedReplica}} after the state check doesn't 
add much.

> Modify ReplicaInfo to remove the assumption that replica metadata and data 
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, 
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the 
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be 
> present on external storages (HDFS-9806). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Chris Douglas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433853#comment-15433853
 ] 

Chris Douglas edited comment on HDFS-10742 at 8/23/16 11:28 PM:


* Is this the intended {{ConcurrentHashMap}} implementation?
{noformat}
+import org.jboss.netty.util.internal.ConcurrentHashMap;
{noformat}
The patch synchronizes on the instance, rather than using {{ConcurrentMap}} 
features. The tracking code could be simplified using those methods.
* Some commented code (e.g., {{+//Thread.dumpStack();}})
* Instead of adding the (unused) {{delay()}} method, consider passing in a 
{{org.apache.hadoop.util.Timer}} instance. This would also make it easier to 
write the unit test.
* There's a comment explaining the importance of using thread-local storage, 
but the current patch neither uses nor requires it.
* Please use the slf4j logger and vararg syntax instead of building strings 
that might be discarded/suppressed.
* If the logger is null, shouldn't this fail? The {{Optional}} wrapper seems 
unnecessary.

On a related note, the virtues of {{AutoCloseableLock}} (HADOOP-13466) are not 
obvious to me. Static analysis tools already catch what it enforces with 
{{AutoCloseable}}. Its interface is almost identical to 
{{java.util.concurrent.locks.Lock}} (admitting extensions like this). It seems 
to add an idiosyncrasy where there is already an idiom (with supporting 
tooling). Is there another advantage?


was (Author: chris.douglas):
* Is this the intended {{ConcurrentHashMap}} implementation?
{noformat}
+import org.jboss.netty.util.internal.ConcurrentHashMap;
{noformat}
The patch synchronizes on the instance, rather than using {{ConcurrentMap}} 
features. The tracking code could be simplified using those methods.
* Some commented code (e.g., {{+//Thread.dumpStack();}})
* Instead of adding the (unused) {{delay()}} method, consider passing in a 
{{org.apache.hadoop.util.Timer}} instance. This would also make it easier to 
write the unit test.
* There's a comment explaining the importance of using thread-local storage, 
but the current patch neither uses nor requires it.
* Please use the slf4j logger and vararg syntax instead of building strings 
that might be discarded/suppressed.
* If the logger is null, shouldn't this fail? The {{Optional}} wrapper seems 
unnecessary.

On a related note, the virtues of {{AutoCloseableLock}} (HADOOP-13466) are not 
obvious to me. Static analysis tools already catch what it enforces with 
{{AutoCloseable}}. Its interface is almost identical to 
{{java.util.concurrent.Lock}} (admitting extensions like this). It seems to add 
an idiosyncrasy where there is already an idiom (with supporting tooling). Is 
there another advantage?

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10742) Measurement of lock held time in FsDatasetImpl

[Chris Douglas (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433853#comment-15433853
 ] 

Chris Douglas commented on HDFS-10742:
--

* Is this the intended {{ConcurrentHashMap}} implementation?
{noformat}
+import org.jboss.netty.util.internal.ConcurrentHashMap;
{noformat}
The patch synchronizes on the instance, rather than using {{ConcurrentMap}} 
features. The tracking code could be simplified using those methods.
* Some commented code (e.g., {{+//Thread.dumpStack();}})
* Instead of adding the (unused) {{delay()}} method, consider passing in a 
{{org.apache.hadoop.util.Timer}} instance. This would also make it easier to 
write the unit test.
* There's a comment explaining the importance of using thread-local storage, 
but the current patch neither uses nor requires it.
* Please use the slf4j logger and vararg syntax instead of building strings 
that might be discarded/suppressed.
* If the logger is null, shouldn't this fail? The {{Optional}} wrapper seems 
unnecessary.

On a related note, the virtues of {{AutoCloseableLock}} (HADOOP-13466) are not 
obvious to me. Static analysis tools already catch what it enforces with 
{{AutoCloseable}}. Its interface is almost identical to 
{{java.util.concurrent.Lock}} (admitting extensions like this). It seems to add 
an idiosyncrasy where there is already an idiom (with supporting tooling). Is 
there another advantage?

> Measurement of lock held time in FsDatasetImpl
> --
>
> Key: HDFS-10742
> URL: https://issues.apache.org/jira/browse/HDFS-10742
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode
>Affects Versions: 3.0.0-alpha2
>Reporter: Chen Liang
>Assignee: Chen Liang
> Attachments: HDFS-10742.001.patch, HDFS-10742.002.patch, 
> HDFS-10742.003.patch
>
>
> This JIRA proposes to measure the time the of lock of {{FsDatasetImpl}} is 
> held by a thread. Doing so will allow us to measure lock statistics.
> This can be done by extending the {{AutoCloseableLock}} lock object in 
> {{FsDatasetImpl}}. In the future we can also consider replacing the lock with 
> a read-write lock.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9696) Garbage snapshot records lingering forever

[Chris Trezzo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433785#comment-15433785
 ] 

Chris Trezzo commented on HDFS-9696:


[~kihwal] Do you think this is worth backporting to branch-2.6? If the unit 
test rewrite is small, it seems to make sense to me.

> Garbage snapshot records lingering forever
> --
>
> Key: HDFS-9696
> URL: https://issues.apache.org/jira/browse/HDFS-9696
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-9696.patch, HDFS-9696.v2.patch
>
>
> We have a cluster where the snapshot feature might have been tested years 
> ago. When the HDFS does not have any snapshot, but I see filediff records 
> persisted in its fsimage.  Since it has been restarted many times and 
> checkpointed over 100 times since then, it must haven been persisted and  
> carried over since then.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10706) Add tool generating FSImage from external store

[Chris Douglas (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Douglas updated HDFS-10706:
-
Attachment: HDFS-10706-HDFS-9806.002.patch

Sorry for the confusion; the HDFS-9806 branch is just a placeholder until 
something is committed. The intent is for issues like HDFS-9809 to get 
committed to trunk, as the base of that branch.

I fast-forwarded the HDFS-9806 branch to make this clearer.

> Add tool generating FSImage from external store
> ---
>
> Key: HDFS-10706
> URL: https://issues.apache.org/jira/browse/HDFS-10706
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: namenode, tools
>Reporter: Chris Douglas
> Attachments: HDFS-10706-HDFS-9806.002.patch, HDFS-10706.001.patch, 
> HDFS-10706.002.patch
>
>
> To experiment with provided storage, this provides a tool to map an external 
> namespace to an FSImage/NN storage. By loading it in a NN, one can access the 
> remote FS using HDFS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10763) Open files can leak permanently due to inconsistent lease update

[Chris Trezzo (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433751#comment-15433751
 ] 

Chris Trezzo commented on HDFS-10763:
-

[~kihwal] do you think this is worth backporting to branch-2.6? It seems like 
the new combined patch is a clean cherry-pick to branch-2.6, but I am not too 
familiar with the differences in snapshot behavior between branch-2.7 and 
branch-2.6.

> Open files can leak permanently due to inconsistent lease update
> 
>
> Key: HDFS-10763
> URL: https://issues.apache.org/jira/browse/HDFS-10763
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.7.3, 2.6.4
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Critical
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10763.br27.patch, 
> HDFS-10763.branch-2.7.supplement.patch, HDFS-10763.branch-2.7.v2.patch, 
> HDFS-10763.patch
>
>
> This can heppen during {{commitBlockSynchronization()}} or a client gives up 
> on closing a file after retries.
> From {{finalizeINodeFileUnderConstruction()}}, the lease is removed first and 
> then the inode is turned into the closed state. But if any block is not in 
> COMPLETE state, 
> {{INodeFile#assertAllBlocksComplete()}} will throw an exception. This will 
> cause the lease is removed from the lease manager, but not from the inode. 
> Since the lease manager does not have a lease for the file, no lease recovery 
> will happen for this file. Moreover, this broken state is persisted and 
> reconstructed through saving and loading of fsimage. Since no replication is 
> scheduled for the blocks for the file, this can cause a data loss and also 
> block decommissioning of datanode.
> The lease cannot be manually recovered either. It fails with
> {noformat}
> ...AlreadyBeingCreatedException): Failed to RECOVER_LEASE /xyz/xyz for user1 
> on
>  0.0.0.1 because the file is under construction but no leases found.
>   at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2950)
> ...
> {noformat}
> When a client retries {{close()}}, the same inconsistent state is created, 
> but it can work in the next time since {{checkLease()}} only looks at the 
> inode, not the lease manager in this case. The close behavior is different if 
> HDFS-8999 is activated by setting 
> {{dfs.namenode.file.close.num-committed-allowed}} to 1 (unlikely) or 2 
> (never). 
> In principle, the under-construction feature of an inode and the lease in the 
> lease manager should never go out of sync. The fix involves two parts.
> 1) Prevent inconsistent lease updates. We can achieve this by calling 
> {{removeLease()}} after checking the block state. 
> 2) Avoid reconstructing inconsistent lease states from a fsimage. 1) alone 
> does not correct the existing inconsistencies surviving through fsimages.  
> This can be done during fsimage loading time by making sure a corresponding 
> lease exists for each inode that are with the underconstruction feature. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.

[Virajith Jalaparti (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433727#comment-15433727
 ] 

Virajith Jalaparti commented on HDFS-10636:
---

Hi [~eddyxu], 

bq.  I know vendors who have already implemented similar solutions based on 
{{FinalizedReplica}}. 
I am trying to understand the use case for {{FinalizedReplica}} you mentioned. 
What properties of {{FinalizedReplica}} do you think we should preserve for 
these implementations to continue working? This question is also related to the 
following. 

bq. Have you considered about making {{LocalReplica}} and {{ProvidedReplica}} 
being subclasses of {{FinalizedReplica}}? 

{{FinalizedReplica}}, as implemented today, does not have any particular 
characteristics other than it's state and type. There do not exist functions in 
{{FinalizedReplica}} that are not in {{ReplicaInfo}}. Is the goal of your 
proposal to have {{FinalizedReplica}} used as a type, and not just state? Do 
you see having to do this for other states as well (RUR, RBW, RWR, TEMPORARY)? 
or having it for {{FinalizedReplica}} is more important? 

One concrete proposal for this can be the following: {{FinalizedReplica}} will 
be an abstract subclass of {{ReplicaInfo}} (overrides 
{{ReplicaInfo::getState()}}). We can then define {{LocalFinalizedReplica}} and 
{{ProvidedFinalizedReplica}}, which can be subclasses of {{FinalizedReplica}}. 
However, this would cause {{LocalFinalizedReplica}} to re-implement the 
functionality of other {{LocalReplica}}s. 
One way to avoid such re-implementation could be to have {{FinalizedReplica}} 
as an interface, with a {{FinalizedReplica::getReplicaInfo()}} function to 
perform all the {{ReplicaInfo}} related functions. Then 
{{LocalFinalizedReplica}} can subclass {{LocalReplica}} and implement the 
{{FinalizedReplica}} interface. 


> Modify ReplicaInfo to remove the assumption that replica metadata and data 
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, 
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the 
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be 
> present on external storages (HDFS-9806). 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9539) libhdfs++: enable default configuration files

[Anatoli Shein (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433674#comment-15433674
 ] 

Anatoli Shein commented on HDFS-9539:
-

The scope of this jira will include updating all examples (and tools) to use 
the correct configs. For tools only one file (tools_common.cpp) possibly will 
need an update.

> libhdfs++: enable default configuration files
> -
>
> Key: HDFS-9539
> URL: https://issues.apache.org/jira/browse/HDFS-9539
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Bob Hansen
>Assignee: Bob Hansen
>
> In the Java implementation of config files, the Hadoop jars included a 
> default core-default and hdfs-default.xml file that provided default values 
> for the run-time configurations.  libhdfs++ should honor that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433628#comment-15433628
 ] 

Kihwal Lee commented on HDFS-10762:
---

+1

> Pass IIP for file status related methods
> 
>
> Key: HDFS-10762
> URL: https://issues.apache.org/jira/browse/HDFS-10762
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10762.1.patch, HDFS-10762.patch
>
>
> The frequently called file status methods will not require path re-resolves 
> if the IIP is passed down the call stack.  The code can be simplified further 
> if the IIP tracks if the original path was a reserved raw path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10789) Route webhdfs through the RPC call queue

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10789:
---
Attachment: HDFS-10789.patch

> Route webhdfs through the RPC call queue
> 
>
> Key: HDFS-10789
> URL: https://issues.apache.org/jira/browse/HDFS-10789
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ipc, webhdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-10789.patch
>
>
> Webhdfs is extremely expensive under load and is not subject to the QoS 
> benefits of the RPC call queue.  HADOOP-13537 provides the basis for routing 
> webhdfs through the call queue to provide unified QoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10608) Include event for AddBlock in Inotify Event Stream

[churro morales (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433618#comment-15433618
 ] 

churro morales commented on HDFS-10608:
---

looked over tests that failed this time around, both tests passed locally and 
seem to be unrelated to this patch.  

[~surendrasingh] how does it look to you?  Thanks again for spending the time 
and reviewing it.

> Include event for AddBlock in Inotify Event Stream
> --
>
> Key: HDFS-10608
> URL: https://issues.apache.org/jira/browse/HDFS-10608
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: churro morales
>Priority: Minor
> Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, 
> HDFS-10608.v2.patch, HDFS-10608.v3.patch
>
>
> It would be nice to have an AddBlockEvent in the INotify pipeline.  Based on 
> discussions from mailing list:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10789) Route webhdfs through the RPC call queue

[Daryn Sharp (JIRA)]

Daryn Sharp created HDFS-10789:
--

 Summary: Route webhdfs through the RPC call queue
 Key: HDFS-10789
 URL: https://issues.apache.org/jira/browse/HDFS-10789
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: ipc, webhdfs
Reporter: Daryn Sharp
Assignee: Daryn Sharp


Webhdfs is extremely expensive under load and is not subject to the QoS 
benefits of the RPC call queue.  HADOOP-13537 provides the basis for routing 
webhdfs through the call queue to provide unified QoS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10788) fsck NullPointerException when it encounters corrupt replicas

[Jeff Field (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Field updated HDFS-10788:
--
Environment: CDH5.5.2, CentOS 6.7  (was: 2.6.0-cdh5.5.2 is the HDFS version 
from the CDH build this cluster is running.)

> fsck NullPointerException when it encounters corrupt replicas
> -
>
> Key: HDFS-10788
> URL: https://issues.apache.org/jira/browse/HDFS-10788
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: namenode
>Affects Versions: 2.6.0
> Environment: CDH5.5.2, CentOS 6.7
>Reporter: Jeff Field
>
> Somehow (I haven't found root cause yet) we ended up with blocks that have 
> corrupt replicas where the replica count is inconsistent between the blockmap 
> and the corrupt replicas map. If we try to hdfs fsck any parent directory 
> that has a child with one of these blocks, fsck will exit with something like 
> this:
> {code}
> $ hdfs fsck /path/to/parent/dir/ | egrep -v '^\.+$'
> Connecting to namenode via http://mynamenode:50070
> FSCK started by bot-hadoop (auth:KERBEROS_SSL) from /10.97.132.43 for path 
> /path/to/parent/dir/ at Tue Aug 23 20:34:58 UTC 2016
> .FSCK 
> ended at Tue Aug 23 20:34:59 UTC 2016 in 1098 milliseconds
> null
> Fsck on path '/path/to/parent/dir/' FAILED
> {code}
> So I start at the top, fscking every subdirectory until I find one or more 
> that fails. Then I do the same thing with those directories (our top level 
> directories all have subdirectories with date directories in them, which then 
> contain the files) and once I find a directory with files in it, I run a 
> checksum of the files in that directory. When I do that, I don't get the name 
> of the file, rather I get:
> checksum: java.lang.NullPointerException
> but since the files are in order, I can figure it out by seeing which file 
> was before the NPE. Once I get to this point, I can see the following in the 
> namenode log when I try to checksum the corrupt file:
> 2016-08-23 20:24:59,627 WARN 
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent 
> number of corrupt replicas for blk_1335893388_1100036319546 blockMap has 0 
> but corrupt replicas map has 1
> 2016-08-23 20:24:59,627 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 23 on 8020, call 
> org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
> 192.168.1.100:47785 Call#1 Retry#0
> java.lang.NullPointerException
> At which point I can delete the file, but it is a very tedious process.
> Ideally, shouldn't fsck be able to emit the name of the file that is the 
> source of the problem - and (if -delete is specified) get rid of the file, 
> instead of exiting without saying why?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10788) fsck NullPointerException when it encounters corrupt replicas

[Jeff Field (JIRA)]

Jeff Field created HDFS-10788:
-

 Summary: fsck NullPointerException when it encounters corrupt 
replicas
 Key: HDFS-10788
 URL: https://issues.apache.org/jira/browse/HDFS-10788
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.6.0
 Environment: 2.6.0-cdh5.5.2 is the HDFS version from the CDH build 
this cluster is running.
Reporter: Jeff Field


Somehow (I haven't found root cause yet) we ended up with blocks that have 
corrupt replicas where the replica count is inconsistent between the blockmap 
and the corrupt replicas map. If we try to hdfs fsck any parent directory that 
has a child with one of these blocks, fsck will exit with something like this:

{code}
$ hdfs fsck /path/to/parent/dir/ | egrep -v '^\.+$'
Connecting to namenode via http://mynamenode:50070
FSCK started by bot-hadoop (auth:KERBEROS_SSL) from /10.97.132.43 for path 
/path/to/parent/dir/ at Tue Aug 23 20:34:58 UTC 2016
.FSCK 
ended at Tue Aug 23 20:34:59 UTC 2016 in 1098 milliseconds
null

Fsck on path '/path/to/parent/dir/' FAILED
{code}

So I start at the top, fscking every subdirectory until I find one or more that 
fails. Then I do the same thing with those directories (our top level 
directories all have subdirectories with date directories in them, which then 
contain the files) and once I find a directory with files in it, I run a 
checksum of the files in that directory. When I do that, I don't get the name 
of the file, rather I get:
checksum: java.lang.NullPointerException

but since the files are in order, I can figure it out by seeing which file was 
before the NPE. Once I get to this point, I can see the following in the 
namenode log when I try to checksum the corrupt file:

2016-08-23 20:24:59,627 WARN 
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Inconsistent number 
of corrupt replicas for blk_1335893388_1100036319546 blockMap has 0 but corrupt 
replicas map has 1
2016-08-23 20:24:59,627 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
23 on 8020, call 
org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 
192.168.1.100:47785 Call#1 Retry#0
java.lang.NullPointerException

At which point I can delete the file, but it is a very tedious process.

Ideally, shouldn't fsck be able to emit the name of the file that is the source 
of the problem - and (if -delete is specified) get rid of the file, instead of 
exiting without saying why?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10784) Implement WebHdfsFileSystem#listStatusIterator

[Andrew Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433558#comment-15433558
 ] 

Andrew Wang commented on HDFS-10784:


Good find Brahma, somehow I forgot about that one. Yea, it does look pretty 
similar, though HDFS-9366 doesn't implement the RemoteIterator interface. I'll 
look at HDFS-9366 to see if I missed anything else though.

> Implement WebHdfsFileSystem#listStatusIterator
> --
>
> Key: HDFS-10784
> URL: https://issues.apache.org/jira/browse/HDFS-10784
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: webhdfs
>Affects Versions: 2.6.4
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-10784.001.patch
>
>
> It would be nice to implement the iterative listStatus in WebHDFS so client 
> apps do not need to buffer the full file list for large directories.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433550#comment-15433550
 ] 

Hadoop QA commented on HDFS-8986:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
34s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
36s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
31s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
23s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
4s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
0s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
10s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 29s{color} | {color:orange} root: The patch generated 4 new + 346 unchanged 
- 10 fixed = 350 total (was 356) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 47 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  6m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
5s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
58s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m  0s{color} 
| {color:red} hadoop-common in the patch failed with JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
15s{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK 
v1.7.0_111. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 

[jira] [Commented] (HDFS-10608) Include event for AddBlock in Inotify Event Stream

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433542#comment-15433542
 ] 

Hadoop QA commented on HDFS-10608:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  9m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 38s{color} | {color:orange} hadoop-hdfs-project: The patch generated 11 new 
+ 193 unchanged - 1 fixed = 204 total (was 194) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
11s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 12s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
24s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 11s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestSafeModeWithStripedFile |
|   | hadoop.hdfs.server.namenode.ha.TestHAAppend |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825108/HDFS-10608.v3.patch |
| JIRA Issue | HDFS-10608 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  cc  |
| uname | Linux 7c07adc6f1d7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8aae8d6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16518/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
 |
| unit | 

[jira] [Commented] (HDFS-9392) Admins support for maintenance state

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1542#comment-1542
 ] 

Hadoop QA commented on HDFS-9392:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 385 
unchanged - 28 fixed = 385 total (was 413) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
54s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m 
35s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 87m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825092/HDFS-9392-4.patch |
| JIRA Issue | HDFS-9392 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 69f0061cc43d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8aae8d6 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16517/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client 
hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16517/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Admins support for maintenance state
> 
>
>

[jira] [Updated] (HDFS-10608) Include event for AddBlock in Inotify Event Stream

[churro morales (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

churro morales updated HDFS-10608:
--
Status: Open  (was: Patch Available)

> Include event for AddBlock in Inotify Event Stream
> --
>
> Key: HDFS-10608
> URL: https://issues.apache.org/jira/browse/HDFS-10608
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: churro morales
>Priority: Minor
> Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, 
> HDFS-10608.v2.patch, HDFS-10608.v3.patch
>
>
> It would be nice to have an AddBlockEvent in the INotify pipeline.  Based on 
> discussions from mailing list:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10608) Include event for AddBlock in Inotify Event Stream

[churro morales (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

churro morales updated HDFS-10608:
--
Status: Patch Available  (was: Open)

> Include event for AddBlock in Inotify Event Stream
> --
>
> Key: HDFS-10608
> URL: https://issues.apache.org/jira/browse/HDFS-10608
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: churro morales
>Priority: Minor
> Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, 
> HDFS-10608.v2.patch, HDFS-10608.v3.patch
>
>
> It would be nice to have an AddBlockEvent in the INotify pipeline.  Based on 
> discussions from mailing list:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10608) Include event for AddBlock in Inotify Event Stream

[churro morales (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

churro morales updated HDFS-10608:
--
Attachment: HDFS-10608.v3.patch

> Include event for AddBlock in Inotify Event Stream
> --
>
> Key: HDFS-10608
> URL: https://issues.apache.org/jira/browse/HDFS-10608
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: churro morales
>Priority: Minor
> Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, 
> HDFS-10608.v2.patch, HDFS-10608.v3.patch
>
>
> It would be nice to have an AddBlockEvent in the INotify pipeline.  Based on 
> discussions from mailing list:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10608) Include event for AddBlock in Inotify Event Stream

[churro morales (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433319#comment-15433319
 ] 

churro morales commented on HDFS-10608:
---

TestDataNodeLifeline and TestDataNodeHotSwapVolumes pass for me locally. 

I did break TestDFSUpgradeFromImage, which I fixed.   I'll upload the patch 
which fixes this issue.  

> Include event for AddBlock in Inotify Event Stream
> --
>
> Key: HDFS-10608
> URL: https://issues.apache.org/jira/browse/HDFS-10608
> Project: Hadoop HDFS
>  Issue Type: Task
>Reporter: churro morales
>Priority: Minor
> Attachments: HDFS-10608.patch, HDFS-10608.v1.patch, 
> HDFS-10608.v2.patch
>
>
> It would be nice to have an AddBlockEvent in the INotify pipeline.  Based on 
> discussions from mailing list:
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201607.mbox/%3C1467743792.4040080.657624289.7BE240AD%40webmail.messagingengine.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-6804) race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433292#comment-15433292
 ] 

Yongjun Zhang commented on HDFS-6804:
-

Thanks for working on [~jojochuang].

For HDFS-10652, I was able to see HDFS-4660 symptom after reverting the 
HDFS-4660 / HDFS-9220 fix. If doing that help reproducing the issue here, maybe 
it's worth a trial.


> race condition between transferring block and appending block causes 
> "Unexpected checksum mismatch exception" 
> --
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Wei-Chiu Chuang
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException: 
> Unexpected checksum mismatch while writing 
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from 
> /192.168.2.101:39495
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Da
> taTransfer: Transmitted 
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad 
> block to NameNode and NameNode marks the replica on the source datanode as 
> corrupt. But actually, the replica on the source datanode is valid. Because 
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433269#comment-15433269
 ] 

Xiaoyu Yao commented on HDFS-10690:
---

[~fenghua_hu], I understand the tradeoff between memory footprint and the 
insert/delete performance here.  Note the ShortCircuitCache#replicaInfoMap is a 
HashMap too, we are O( n) worst case and O(1) on average anyway for 
ref()/unref() case. So I don't think that will change the time complexity of 
ref()/unref() case. 

After taking another look at the get() usage issue you mentioned earlier, I 
don't think we need the extra indexOf(). A LinkedMap is extended from 
AbstractLinkedMap, which extended from AbstractHashedMap. Because of that, we 
are calling into the AbstractHashedMap#get(Object Key) here according to 
https://commons.apache.org/proper/commonscollections/jacoco/org.apache.commons.collections4.map/AbstractHashedMap.java.html.
 This will still be O(1) on average and O(n) worst case. 

I would appreciate if you could post the result of the simpler approach that I 
proposed under the same test environment. If the difference is significant, we 
can look at alternatives as you suggested. Let me know your thoughts. Thanks!


> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Xiaoyu Yao (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433269#comment-15433269
 ] 

Xiaoyu Yao edited comment on HDFS-10690 at 8/23/16 5:40 PM:


[~fenghua_hu], I understand the tradeoff between memory footprint and the 
insert/delete performance here.  Note the ShortCircuitCache#replicaInfoMap is a 
HashMap too, we are O( n) worst case and O(1) on average anyway for 
ref()/unref() case. So I don't think that will change the time complexity of 
ref()/unref() case. 

After taking another look at the get() usage issue you mentioned earlier, I 
don't think we need the extra indexOf(). A LinkedMap is extended from 
AbstractLinkedMap, which extended from AbstractHashedMap. Because of that, we 
are calling into the AbstractHashedMap#get(Object Key) here according to 
https://commons.apache.org/proper/commonscollections/jacoco/org.apache.commons.collections4.map/AbstractHashedMap.java.html.
 This will still be O(1) on average and O( n) worst case. 

I would appreciate if you could post the result of the simpler approach that I 
proposed under the same test environment. If the difference is significant, we 
can look at alternatives as you suggested. Let me know your thoughts. Thanks!



was (Author: xyao):
[~fenghua_hu], I understand the tradeoff between memory footprint and the 
insert/delete performance here.  Note the ShortCircuitCache#replicaInfoMap is a 
HashMap too, we are O( n) worst case and O(1) on average anyway for 
ref()/unref() case. So I don't think that will change the time complexity of 
ref()/unref() case. 

After taking another look at the get() usage issue you mentioned earlier, I 
don't think we need the extra indexOf(). A LinkedMap is extended from 
AbstractLinkedMap, which extended from AbstractHashedMap. Because of that, we 
are calling into the AbstractHashedMap#get(Object Key) here according to 
https://commons.apache.org/proper/commonscollections/jacoco/org.apache.commons.collections4.map/AbstractHashedMap.java.html.
 This will still be O(1) on average and O(n) worst case. 

I would appreciate if you could post the result of the simpler approach that I 
proposed under the same test environment. If the difference is significant, we 
can look at alternatives as you suggested. Let me know your thoughts. Thanks!


> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-6804) race condition between transferring block and appending block causes "Unexpected checksum mismatch exception"

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-6804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433239#comment-15433239
 ] 

Wei-Chiu Chuang commented on HDFS-6804:
---

I have been working on this for quite some time. However, I have not been able 
to reproduce it following the steps described by [~wangg23]. I am thinking 
whether it is the same bug as HDFS-4660. If so, it could explain why I couldn't 
reproduce it because it's fixed.

> race condition between transferring block and appending block causes 
> "Unexpected checksum mismatch exception" 
> --
>
> Key: HDFS-6804
> URL: https://issues.apache.org/jira/browse/HDFS-6804
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.2.0
>Reporter: Gordon Wang
>Assignee: Wei-Chiu Chuang
>
> We found some error log in the datanode. like this
> {noformat}
> 2014-07-22 01:49:51,338 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Ex
> ception for BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248
> java.io.IOException: Terminating due to a checksum error.java.io.IOException: 
> Unexpected checksum mismatch while writing 
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997_9248 from 
> /192.168.2.101:39495
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:536)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:703)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:575)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
> at 
> org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> While on the source datanode, the log says the block is transmitted.
> {noformat}
> 2014-07-22 01:49:50,805 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> Da
> taTransfer: Transmitted 
> BP-2072804351-192.168.2.104-1406008383435:blk_1073741997
> _9248 (numBytes=16188152) to /192.168.2.103:50010
> {noformat}
> When the destination datanode gets the checksum mismatch, it reports bad 
> block to NameNode and NameNode marks the replica on the source datanode as 
> corrupt. But actually, the replica on the source datanode is valid. Because 
> the replica can pass the checksum verification.
> In all, the replica on the source data is wrongly marked as corrupted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission

[John Zhuge (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433238#comment-15433238
 ] 

John Zhuge commented on HDFS-10650:
---

Done, [~rchiang] and [~xiaochen].

> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory 
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10650.001.patch, HDFS-10650.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a 
> directory.
> {code:java}
>   public boolean mkdirs(String src, FsPermission permission,
>   boolean createParent) throws IOException {
> if (permission == null) {
>   permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
>   public boolean primitiveMkdir(String src, FsPermission absPermission, 
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
>   absPermission = 
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> } 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission

[John Zhuge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Zhuge updated HDFS-10650:
--
Release Note: If the caller does not supply a permission, DFSClient#mkdirs 
and DFSClient#primitiveMkdir will create a new directory with the default 
directory permission 00777 instead of 00666.

> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory 
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10650.001.patch, HDFS-10650.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a 
> directory.
> {code:java}
>   public boolean mkdirs(String src, FsPermission permission,
>   boolean createParent) throws IOException {
> if (permission == null) {
>   permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
>   public boolean primitiveMkdir(String src, FsPermission absPermission, 
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
>   absPermission = 
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> } 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-9392) Admins support for maintenance state

[Ming Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ming Ma updated HDFS-9392:
--
Attachment: HDFS-9392-4.patch

New patch with additional checkstyle fixes.

> Admins support for maintenance state
> 
>
> Key: HDFS-9392
> URL: https://issues.apache.org/jira/browse/HDFS-9392
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Ming Ma
> Attachments: HDFS-9392-2.patch, HDFS-9392-3.patch, HDFS-9392-4.patch, 
> HDFS-9392.patch
>
>
> This is to allow admins to put nodes into maintenance state with optional 
> timeout value as well as take nodes out of maintenance state. Likely we will 
> leverage what we come up in https://issues.apache.org/jira/browse/HDFS-9005.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433142#comment-15433142
 ] 

Fenghua Hu edited comment on HDFS-10690 at 8/23/16 4:43 PM:


I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+   
  +-+
 |   Replica 1  |-Next> |Replica  2  |-Next> |  
  Replica  3  |
 |   Replica 1  |<-Prev |Replica  2  |<-Prev |  
  Replica  3  |
+++-+   
 +-+


We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replicas are doubly linked in order of insertion time. The youngest is 
always at the head of the linked list, and the eldest is always at the tail.  
Removing the entries between the head and the tail doesn't need any lookup, 
because the replica knows its position in linked list by next and prev, thus 
remove is simple: change it's precedessor's and its succssor's next and prev. 
The order of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.


was (Author: fenghua_hu):
I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+   
 +-+
  >|   Replica 1  |-Next> |Replica  2  |-Next> |
Replica  3  |
 <-|   Replica 1  |<-Prev |Replica  2  |<-Prev |
Replica  3  |
+++-+   
 +-+


We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: 

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433142#comment-15433142
 ] 

Fenghua Hu edited comment on HDFS-10690 at 8/23/16 4:41 PM:


I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+   
 +-+
  >|   Replica 1  |-Next> |Replica  2  |-Next> |
Replica  3  |
 <-|   Replica 1  |<-Prev |Replica  2  |<-Prev |
Replica  3  |
+++-+   
 +-+


We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.


was (Author: fenghua_hu):
I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+   
 +-+
 || |  
| |  |
  >|   Replica 1  |-Next> |Replica  2  |-Next> |
Replica  3  |
 || |  
| |  |
 <-||<-Prev |  
|<-Prev |  |
++ +-+  
  +-+


We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.

> Optimize insertion/removal of 

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433142#comment-15433142
 ] 

Fenghua Hu edited comment on HDFS-10690 at 8/23/16 4:38 PM:


I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+   
 +-+
 || |  
| |  |
  >|   Replica 1  |-Next> |Replica  2  |-Next> |
Replica  3  |
 || |  
| |  |
 <-||<-Prev |  
|<-Prev |  |
++ +-+  
  +-+


We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.


was (Author: fenghua_hu):
I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+
 |Replica 1 | |Replica  2 |
+++-+
  >| Next|-> | Next |--->...
+++-+
 <-| Prev|<-| Prev |<--...
+++-+
 || | |
+++-+

We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ 

[jira] [Updated] (HDFS-10786) Erasure Coding: Add removeErasureCodingPolicy API

[Zhe Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhe Zhang updated HDFS-10786:
-
Labels: hdfs-ec-3.0-must-do  (was: )

> Erasure Coding: Add removeErasureCodingPolicy API
> -
>
> Key: HDFS-10786
> URL: https://issues.apache.org/jira/browse/HDFS-10786
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xinwei Qin 
>  Labels: hdfs-ec-3.0-must-do
>
> HDFS-7859 has developed addErasureCodingPolicy API to add some user-added 
> Erasure Coding policies, and as discussed in HDFS-7859, we should also add 
> removeErasureCodingPolicy API to support removing some user-added Erasure 
> Coding Polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433142#comment-15433142
 ] 

Fenghua Hu edited comment on HDFS-10690 at 8/23/16 4:32 PM:


I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+
 |Replica 1 | |Replica  2 |
+++-+
  >| Next|-> | Next |--->...
+++-+
 <-| Prev|<-| Prev |<--...
+++-+
 || | |
+++-+

We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.


was (Author: fenghua_hu):
I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+
 |Replica 1 | |Replica  2 |
+++-+
..>| Next|->| Next |--->...
+++-+
.<-| Prev|<-| Prev |<--...
+++-+
 || | |
+++-+

We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.

> Optimize insertion/removal of 

[jira] [Comment Edited] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433142#comment-15433142
 ] 

Fenghua Hu edited comment on HDFS-10690 at 8/23/16 4:30 PM:


I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(log n) operation. To improve 
it, we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+
 |Replica 1 | |Replica  2 |
+++-+
..>| Next|->| Next |--->...
+++-+
.<-| Prev|<-| Prev |<--...
+++-+
 || | |
+++-+

We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.


was (Author: fenghua_hu):
I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(n) operation. To improve it, 
we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+
 |Replica 1 | |Replica  2 |
..>| Next --|->|Next--- |--->...
.<-|Prev|<-|Prev |<--...
 ||| |
++   ++

We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: 

[jira] [Commented] (HDFS-10781) libhdfs++: redefine NN timeout to be "time without a response"

[Anatoli Shein (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433143#comment-15433143
 ] 

Anatoli Shein commented on HDFS-10781:
--

Current hack that we are using in the code is:

{code}
//TODO: HDFS-9539 - until then we increase the time-out to allow all recursive 
async calls to finish
options.rpc_timeout = std::numeric_limits::max();
{code}

This is done in all examples and tools that are recursive.

> libhdfs++: redefine NN timeout to be "time without a response"
> --
>
> Key: HDFS-10781
> URL: https://issues.apache.org/jira/browse/HDFS-10781
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Bob Hansen
>
> In the find tool, we submit a zillion requests to the NameNode 
> asynchronously.  As the queue on the NameNode grows, the time to response for 
> each individual message will increase.  In the find tool, we were eventually 
> getting timeouts on requests, even though the NN was respoinding as fast as 
> its little feet could carry it.
> I propose that we should redefine timeouts to be on a per-connection basis 
> rather than per-request.  If a client has an outstanding request to the NN 
> but hasn't gotten a response back within n msec, it should declare the 
> connection dead and retry.  As long as the NameNode is being responsive to 
> the best of its ability and providing data, we will not declare the link dead.
> One potential for Failure of Least Astonishment here is that it will mean any 
> particular request from a client cannot be depended on to get a positive or 
> negative response within a fixed amount of time, but I think that may be a 
> good trade to make.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10690) Optimize insertion/removal of replica in ShortCircuitCache.java

[Fenghua Hu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433142#comment-15433142
 ] 

Fenghua Hu commented on HDFS-10690:
---

I would like to explain the solution here.

Currently, TreeMap is used to track all the ShortCircuitReplica entries in the 
order of being inserted. These entries could be removed from TreeMap in two 
cases. The first is when the entry is accessed again, it will be removed from 
TreeMap. Please note that the entry could be anyone in the TreeMap. The other 
is when the entry is evicted due to treemap size limitation, in this case, only 
the eldest entry will be removed.

Removal is a costly operation for the first case, because looking up 
ShortCircuitReplica is needed, in TreeMap, it's O(n) operation. To improve it, 
we design a new data structure LruList, which entirely eliminates costly 
look-up operation. 
+++-+
 |Replica 1 | |Replica  2 |
..>| Next --|->|Next--- |--->...
.<-|Prev|<-|Prev |<--...
 ||| |
++   ++

We introduced two references in ShortCircuitReplica objects. Reference Next 
points to the elder ShortCircuitReplica and Prev points to the younger one. All 
the replica is doubly linked in order of insertion time. The youngest is always 
at the head of the linked list, and the eldest is always at the tail.  Removing 
the entries between the head and the tail doesn't need any lookup, because the 
replica knows its position in linked list by next and prev, thus remove is 
simple: change it's precedessor's and its succssor's next and prev. The order 
of operation is always O(1). 

For insertion, the youngest entry is always be added to the head, thus the 
operation is also O(1).

Existing classes, including LinkedHashMap, LinkedMap, can't provide O(1) 
operation for insertion/lookup/removal.

Here comes a brief test result:

Run GET queries with 64 YCSB processes for 30 minutes, record the QPS for each 
process。
Total QPS:
w/o patch: 95K
w/ patch: 135K

The performance gain is (135 - 95) / 95 = 42%.

Suggestions/comments are very welcomed.

> Optimize insertion/removal of replica in ShortCircuitCache.java
> ---
>
> Key: HDFS-10690
> URL: https://issues.apache.org/jira/browse/HDFS-10690
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 3.0.0-alpha2
>Reporter: Fenghua Hu
>Assignee: Fenghua Hu
> Attachments: HDFS-10690.001.patch, HDFS-10690.002.patch, 
> ShortCircuitCache_LinkedMap.patch
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> Currently in ShortCircuitCache, two TreeMap objects are used to track the 
> cached replicas.
> private final TreeMap evictable = new TreeMap<>();
> private final TreeMap evictableMmapped = new 
> TreeMap<>();
> TreeMap employs Red-Black tree for sorting. This isn't an issue when using 
> traditional HDD. But when using high-performance SSD/PCIe Flash, the cost 
> inserting/removing an entry  becomes considerable.
> To mitigate it, we designed a new list-based for replica tracking.
> The list is a double-linked FIFO. FIFO is time-based, thus insertion is a 
> very low cost operation. On the other hand, list is not lookup-friendly. To 
> address this issue, we introduce two references into ShortCircuitReplica 
> object.
> ShortCircuitReplica next = null;
> ShortCircuitReplica prev = null;
> In this way, lookup is not needed when removing a replica from the list. We 
> only need to modify its predecessor's and successor's references in the lists.
> Our tests showed up to 15-50% performance improvement when using PCIe flash 
> as storage media.
> The original patch is against 2.6.4, now I am porting to Hadoop trunk, and 
> patch will be posted soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433115#comment-15433115
 ] 

Hadoop QA commented on HDFS-7859:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 41s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new + 
1118 unchanged - 1 fixed = 1119 total (was 1119) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
33s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client generated 1 new 
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 58s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 18s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-client |
|  |  Class 
org.apache.hadoop.hdfs.protocol.datatransfer.ReplaceDatanodeOnFailure$Policy 
defines non-transient non-serializable instance field condition  In 
ReplaceDatanodeOnFailure.java:instance field condition  In 
ReplaceDatanodeOnFailure.java |
| Failed junit tests | 
hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825071/HDFS-7859.007.patch |
| JIRA Issue | HDFS-7859 |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  javadoc  
mvninstall  findbugs  checkstyle  |
| uname | Linux 83a4efe3a9e4 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (HDFS-9745) TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433048#comment-15433048
 ] 

Xiao Chen commented on HDFS-9745:
-

Woot. Thanks [~jlowe]!

> TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
> 
>
> Key: HDFS-9745
> URL: https://issues.apache.org/jira/browse/HDFS-9745
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Fix For: 2.8.0, 2.7.4
>
> Attachments: HDFS-9745.01.patch
>
>
> TestSecureNNWithQJM#testSecureMode fails intermittently. For most of the 
> case, it timeouts.
> In a 0.5%~1% probability, it fails with a more sophisticated error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10787) libhdfs++: hdfs_configuration and configuration_loader should be accessible from our public API

[Anatoli Shein (JIRA)]

Anatoli Shein created HDFS-10787:


 Summary: libhdfs++: hdfs_configuration and configuration_loader 
should be accessible from our public API
 Key: HDFS-10787
 URL: https://issues.apache.org/jira/browse/HDFS-10787
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Anatoli Shein


Currently, libhdfspp examples and tools all have this:
#include "hdfspp/hdfspp.h"
#include "common/hdfs_configuration.h"
#include "common/configuration_loader.h"

This is done in order to read configs and connect. We want  hdfs_configuration 
and configuration_loader to be accessible just by including our hdfspp.h. One 
way to achieve that would be to create a builder, and would include the above 
libraries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10786) Erasure Coding: Add removeErasureCodingPolicy API

[Xinwei Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xinwei Qin  updated HDFS-10786:
---
Description: HDFS-7859 has developed addErasureCodingPolicy API to add some 
user-added Erasure Coding policies, and as discussed in HDFS-7859, we should 
also add removeErasureCodingPolicy API to support removing some user-added 
Erasure Coding Polices.  (was: HDFS-7859 has developed addErasureCodingPolicy 
API to add some user-added Erasure Coding policies, we should also add 
removeErasureCodingPolicy API to support removing some user-added Erasure 
Coding Polices.)

> Erasure Coding: Add removeErasureCodingPolicy API
> -
>
> Key: HDFS-10786
> URL: https://issues.apache.org/jira/browse/HDFS-10786
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xinwei Qin 
>
> HDFS-7859 has developed addErasureCodingPolicy API to add some user-added 
> Erasure Coding policies, and as discussed in HDFS-7859, we should also add 
> removeErasureCodingPolicy API to support removing some user-added Erasure 
> Coding Polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode

[Xinwei Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432993#comment-15432993
 ] 

Xinwei Qin  commented on HDFS-7859:
---

Hi, [~zhz],
  Have updated the patch with your comments and fixed some UT failure, pls 
review.

> Erasure Coding: Persist erasure coding policies in NameNode
> ---
>
> Key: HDFS-7859
> URL: https://issues.apache.org/jira/browse/HDFS-7859
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Xinwei Qin 
>  Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do
> Attachments: HDFS-7859-HDFS-7285.002.patch, 
> HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch, 
> HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, 
> HDFS-7859.005.patch, HDFS-7859.006.patch, HDFS-7859.007.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we 
> persist EC schemas in NameNode centrally and reliably, so that EC zones can 
> reference them by name efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10786) Erasure Coding: Add removeErasureCodingPolicy API

[Xinwei Qin (JIRA)]

Xinwei Qin  created HDFS-10786:
--

 Summary: Erasure Coding: Add removeErasureCodingPolicy API
 Key: HDFS-10786
 URL: https://issues.apache.org/jira/browse/HDFS-10786
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Xinwei Qin 


HDFS-7859 has developed addErasureCodingPolicy API to add some user-added 
Erasure Coding policies, we should also add removeErasureCodingPolicy API to 
support removing some user-added Erasure Coding Polices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10762) Pass IIP for file status related methods

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432976#comment-15432976
 ] 

Hadoop QA commented on HDFS-10762:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
8s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 1 new + 105 unchanged - 2 fixed = 106 total (was 107) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 58m 
59s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 30s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825053/HDFS-10762.1.patch |
| JIRA Issue | HDFS-10762 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2d44811d8e89 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / f0efea4 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16512/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16512/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16512/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Pass IIP for file status related methods
> 
>
> Key: HDFS-10762
> URL: https://issues.apache.org/jira/browse/HDFS-10762
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0-alpha2
>

[jira] [Updated] (HDFS-9745) TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated HDFS-9745:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 2.7.4
   2.8.0
   Status: Resolved  (was: Patch Available)

Thanks, [~xiaochen]!  I committed this to trunk, branch-2, branch-2.8, and 
branch-2.7.

> TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
> 
>
> Key: HDFS-9745
> URL: https://issues.apache.org/jira/browse/HDFS-9745
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Fix For: 2.8.0, 2.7.4
>
> Attachments: HDFS-9745.01.patch
>
>
> TestSecureNNWithQJM#testSecureMode fails intermittently. For most of the 
> case, it timeouts.
> In a 0.5%~1% probability, it fails with a more sophisticated error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10679) libhdfs++: Implement parallel find with wildcards tool

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432974#comment-15432974
 ] 

Bob Hansen commented on HDFS-10679:
---

[~anatoli.shein]: this looks very usesful.  I like demonstrating both sync and 
async find.  Nice.

FS::Find:
* Async - Callback deliver results with a const std::vector &, not a 
shared_ptr.  This is a signal to the consumer to use the data delivered during 
the callback, but don't use the passed-in container.
* Likewise, the synchronous call should take a non-const std::vector 
* for an output parameter, signaling to the consumer that we are going to 
mutate their input vector
* We need a very clear threading model.  Will the handler be called 
concurrently from multiple threads (currently, yes.  If we ever get on asio 
fibers, we should make it a no, because we love our consumers)
* We're doing a _lot_ of dynamic memory allocation during recursion.  Could we 
restructure things a little to not copy the entirety of the FindState and 
RecursionState on each call?  It appears that they each have one element that 
is being updated for each recursive call
* We need to hold the lock while incrementing the recursion_counter also
* If the handler returns false (don't want more) at the end of the function, do 
we do anything to prevent more from being delivered?  Should we push that into 
the shared find_state and bail out for any subsequent NN responses?


find.cpp: 
* Like the cat examples, simplify as much as possible.  Nuke URI parsing, etc.
* Expand smth_found to something_found to prevent confusion (especially in an 
example)
* We have race conditions if one thread is outputting the previous block while 
another thread gets a final block (or error).  

FS::GetFileInfo should populate the full_path member also

> libhdfs++: Implement parallel find with wildcards tool
> --
>
> Key: HDFS-10679
> URL: https://issues.apache.org/jira/browse/HDFS-10679
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10679.HDFS-8707.000.patch, 
> HDFS-10679.HDFS-8707.001.patch, HDFS-10679.HDFS-8707.002.patch, 
> HDFS-10679.HDFS-8707.003.patch, HDFS-10679.HDFS-8707.004.patch, 
> HDFS-10679.HDFS-8707.005.patch, HDFS-10679.HDFS-8707.006.patch, 
> HDFS-10679.HDFS-8707.007.patch, HDFS-10679.HDFS-8707.008.patch, 
> HDFS-10679.HDFS-8707.009.patch, HDFS-10679.HDFS-8707.010.patch, 
> HDFS-10679.HDFS-8707.011.patch, HDFS-10679.HDFS-8707.012.patch, 
> HDFS-10679.HDFS-8707.013.patch
>
>
> The find tool will issue the GetListing namenode operation on a given 
> directory, and filter the results using posix globbing library.
> If the recursive option is selected, for each returned entry that is a 
> directory the tool will issue another asynchronous call GetListing and repeat 
> the result processing in a recursive fashion.
> One implementation issue that needs to be addressed is the way how results 
> are returned back to the user: we can either buffer the results and return 
> them to the user in bulk, or we can return results continuously as they 
> arrive. While buffering would be an easier solution, returning results as 
> they arrive would be more beneficial to the user in terms of performance, 
> since the result processing can start as soon as the first results arrive 
> without any delay. In order to do that we need the user to use a loop to 
> process arriving results, and we need to send a special message back to the 
> user when the search is over.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9745) TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432964#comment-15432964
 ] 

Hudson commented on HDFS-9745:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10329 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10329/])
HDFS-9745. TestSecureNNWithQJM#testSecureMode sometimes fails with (jlowe: rev 
126d165efd80e266a8309241f3cf059e358f5019)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/TestSecureNNWithQJM.java


> TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
> 
>
> Key: HDFS-9745
> URL: https://issues.apache.org/jira/browse/HDFS-9745
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Fix For: 2.8.0, 2.7.4
>
> Attachments: HDFS-9745.01.patch
>
>
> TestSecureNNWithQJM#testSecureMode fails intermittently. For most of the 
> case, it timeouts.
> In a 0.5%~1% probability, it fails with a more sophisticated error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8224) Schedule a block for scanning if its metadata file is corrupt

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432949#comment-15432949
 ] 

Wei-Chiu Chuang commented on HDFS-8224:
---

Committed this to 2.7.
Branch 2.6 uses the old implementation of DataBlockScanner, so this patch is 
not compatible.

> Schedule a block for scanning if its metadata file is corrupt
> -
>
> Key: HDFS-8224
> URL: https://issues.apache.org/jira/browse/HDFS-8224
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-8224-branch-2.7.patch, HDFS-8224-branch-2.patch, 
> HDFS-8224-trunk-1.patch, HDFS-8224-trunk-2.patch, HDFS-8224-trunk-3.patch, 
> HDFS-8224-trunk.patch
>
>
> This happened in our 2.6 cluster.
> One of the block and its metadata file were corrupted.
> The disk was healthy in this case.
> Only the block was corrupt.
> Namenode tried to copy that block to another datanode but failed with the 
> following stack trace:
> 2015-04-20 01:04:04,421 
> [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4] WARN 
> datanode.DataNode: DatanodeRegistration(a.b.c.d, 
> datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7, infoPort=1006, 
> infoSecurePort=0, ipcPort=8020, 
> storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
>  to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to 
> a1.b1.c1.d1:1004 got 
> java.io.IOException: Could not create DataChecksum of type 0 with 
> bytesPerChecksum 0
> at 
> org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:287)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
> at java.lang.Thread.run(Thread.java:722)
> The following catch block in DataTransfer#run method will treat every 
> IOException as disk error fault and run disk errror
> {noformat}
> catch (IOException ie) {
> LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
> targets[0] + " got ", ie);
> // check if there are any disk problem
> checkDiskErrorAsync();
>   } 
> {noformat}
> This block was never scanned by BlockPoolSliceScanner otherwise it would have 
> reported as corrupt block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9745) TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432939#comment-15432939
 ] 

Hadoop QA commented on HDFS-9745:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} HDFS-9745 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12785890/HDFS-9745.01.patch |
| JIRA Issue | HDFS-9745 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16514/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
> 
>
> Key: HDFS-9745
> URL: https://issues.apache.org/jira/browse/HDFS-9745
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HDFS-9745.01.patch
>
>
> TestSecureNNWithQJM#testSecureMode fails intermittently. For most of the 
> case, it timeouts.
> In a 0.5%~1% probability, it fails with a more sophisticated error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432937#comment-15432937
 ] 

Hadoop QA commented on HDFS-8986:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  7m 
13s{color} | {color:red} Docker failed to build yetus/hadoop:b59b8b7. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825068/HDFS-8986.branch-2.patch
 |
| JIRA Issue | HDFS-8986 |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16513/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add option to -du to calculate directory space usage excluding snapshots
> 
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch, 
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, 
> HDFS-8986.06.patch, HDFS-8986.07.patch, HDFS-8986.branch-2.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its 
> children), the report includes space consumed by blocks that are only present 
> in the snapshots. This is confusing for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related 
> disk usage in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-7859) Erasure Coding: Persist erasure coding policies in NameNode

[Xinwei Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xinwei Qin  updated HDFS-7859:
--
Attachment: HDFS-7859.007.patch

> Erasure Coding: Persist erasure coding policies in NameNode
> ---
>
> Key: HDFS-7859
> URL: https://issues.apache.org/jira/browse/HDFS-7859
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Xinwei Qin 
>  Labels: BB2015-05-TBR, hdfs-ec-3.0-must-do
> Attachments: HDFS-7859-HDFS-7285.002.patch, 
> HDFS-7859-HDFS-7285.002.patch, HDFS-7859-HDFS-7285.003.patch, 
> HDFS-7859.001.patch, HDFS-7859.002.patch, HDFS-7859.004.patch, 
> HDFS-7859.005.patch, HDFS-7859.006.patch, HDFS-7859.007.patch
>
>
> In meetup discussion with [~zhz] and [~jingzhao], it's suggested that we 
> persist EC schemas in NameNode centrally and reliably, so that EC zones can 
> reference them by name efficiently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9745) TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432932#comment-15432932
 ] 

Jason Lowe commented on HDFS-9745:
--

Seeing occasional timeouts for this test as well.

+1 lgtm.  Committing this.

> TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts
> 
>
> Key: HDFS-9745
> URL: https://issues.apache.org/jira/browse/HDFS-9745
> Project: Hadoop HDFS
>  Issue Type: Test
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Minor
> Attachments: HDFS-9745.01.patch
>
>
> TestSecureNNWithQJM#testSecureMode fails intermittently. For most of the 
> case, it timeouts.
> In a 0.5%~1% probability, it fails with a more sophisticated error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10679) libhdfs++: Implement parallel find with wildcards tool

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432919#comment-15432919
 ] 

Hadoop QA commented on HDFS-10679:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
36s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
38s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
39s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
15s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
9s{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
11s{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
39s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  5m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
36s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  5m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
6s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
8s{color} | {color:green} the patch passed with JDK v1.7.0_101 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m  
4s{color} | {color:green} hadoop-hdfs-native-client in the patch passed with 
JDK v1.7.0_101. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0cf5e66 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825054/HDFS-10679.HDFS-8707.013.patch
 |
| JIRA Issue | HDFS-10679 |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  javadoc  
mvninstall  |
| uname | Linux 14faf7b09832 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | HDFS-8707 / 4a74bc4 |
| Default Java | 1.7.0_101 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_101 
/usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 |
| JDK v1.7.0_101  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16511/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U: 
hadoop-hdfs-project/hadoop-hdfs-native-client |
| 

[jira] [Updated] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Xiao Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xiao Chen updated HDFS-8986:

Attachment: HDFS-8986.branch-2.patch

Thanks Wei-Chiu.

HADOOP-11666 reverted an incompatible change, which caused backport conflicts 
on this patch. This patch itself is compatible IMO, attaching a patch For 
branch-2.

> Add option to -du to calculate directory space usage excluding snapshots
> 
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch, 
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, 
> HDFS-8986.06.patch, HDFS-8986.07.patch, HDFS-8986.branch-2.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its 
> children), the report includes space consumed by blocks that are only present 
> in the snapshots. This is confusing for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related 
> disk usage in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-8224) Schedule a block for scanning if its metadata file is corrupt

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang updated HDFS-8224:
--
Fix Version/s: 2.7.4

> Schedule a block for scanning if its metadata file is corrupt
> -
>
> Key: HDFS-8224
> URL: https://issues.apache.org/jira/browse/HDFS-8224
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.8.0, 2.9.0, 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-8224-branch-2.7.patch, HDFS-8224-branch-2.patch, 
> HDFS-8224-trunk-1.patch, HDFS-8224-trunk-2.patch, HDFS-8224-trunk-3.patch, 
> HDFS-8224-trunk.patch
>
>
> This happened in our 2.6 cluster.
> One of the block and its metadata file were corrupted.
> The disk was healthy in this case.
> Only the block was corrupt.
> Namenode tried to copy that block to another datanode but failed with the 
> following stack trace:
> 2015-04-20 01:04:04,421 
> [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4] WARN 
> datanode.DataNode: DatanodeRegistration(a.b.c.d, 
> datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7, infoPort=1006, 
> infoSecurePort=0, ipcPort=8020, 
> storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
>  to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to 
> a1.b1.c1.d1:1004 got 
> java.io.IOException: Could not create DataChecksum of type 0 with 
> bytesPerChecksum 0
> at 
> org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:287)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
> at java.lang.Thread.run(Thread.java:722)
> The following catch block in DataTransfer#run method will treat every 
> IOException as disk error fault and run disk errror
> {noformat}
> catch (IOException ie) {
> LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
> targets[0] + " got ", ie);
> // check if there are any disk problem
> checkDiskErrorAsync();
>   } 
> {noformat}
> This block was never scanned by BlockPoolSliceScanner otherwise it would have 
> reported as corrupt block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432872#comment-15432872
 ] 

Bob Hansen commented on HDFS-10754:
---

One more thing...

Should we add a logger to the tools?  If we get a NN fault and issue a warning, 
will it get logged to stderr?  Is that the desired behavior?

> libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, 
> hdfs_chown, and hdfs_chmod
> 
>
> Key: HDFS-10754
> URL: https://issues.apache.org/jira/browse/HDFS-10754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10754.HDFS-8707.000.patch, 
> HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, 
> HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, 
> HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, 
> HDFS-10754.HDFS-8707.007.patch, HDFS-10754.HDFS-8707.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10719) In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432866#comment-15432866
 ] 

Hadoop QA commented on HDFS-10719:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch 
generated 3 new + 14 unchanged - 0 fixed = 17 total (was 14) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 55s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m  0s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
|   | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12824853/HDFS-10719-3.patch |
| JIRA Issue | HDFS-10719 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux bf430ba7ac44 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / f0efea4 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16510/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16510/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16510/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16510/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically 

[jira] [Commented] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod

[Bob Hansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432868#comment-15432868
 ] 

Bob Hansen commented on HDFS-10754:
---

This is lots of good work, [~anatoli.shein].  I have a handful of mostly minor 
things I might propose:

cat:
* Why are we including "google/protobuf/stubs/common.h?
* Comment says "wrapping fs in unique_ptr", but then we wrap it in a shared_ptr.
* We shouldn't check the port on defaultFS before reporting error.  The 
defaultFS might point to an HA name.  Just check that defaultFS isn't empty, 
and report it if it is.
* For simplicity, cat should just use file->Read rather than PositionRead

gendirs:
* As an example, gendirs should _definitely_ not need to use 
fs/namenode_operations
* It should not need to use common/* either; we should push the configuration 
stuff into hdfspp/ (but perhaps that can be a different bug)
* Again, why include the protobuf header?
* We generally don't make values on the stack const (e.g. path, depth, fanout). 
 It's not wrong, just generally redundant (unless it's important they be const 
for some reason)
* Put a comment on setting the timeout referring to HDFS-10781 (and vice-versa)

configuration_loader.cc:
* Add a comment on why we're calling setDefaultSearchPath in the ctor

configuration_loader.h:
* I might make the comment something along the lines of "Creates a 
configuration loader with the default search path ().  If you want to 
explicitly set the entire search path, call ClearSearchPath() first"
* Filesystem.cc: can SetOwner/SetPermission be written as a call to 
::Find(recursive=true) with the SetOwner/SetPerimission implemented in the 
callback?  Then we wouldn't need three separate implementations of the 
recursion logic
* Does recursive SetOwner/SetPermissions accept globs both for the recursive 
and non-recursive versions?  We should be consistent.  Perhaps 
SetOwner(explicit filename, fast) and SetOwner(glob/recursive, slower) should 
be different methods

Tools impl:
* Make a usage() function that prints out the usage of the tool.  Call it if 
"--help", "-h", or a parameter problem occurs.
* Keen gendirs in the examples.  I don't think we need a tool version of it.

Include a comment on HDFS-9539 to fix up the tools and examples as part of the 
scope.

hdfsNewBuilderFromDirectory (in hdfs.cc) should call ClearSearchPath rather 
than inheriting the default.  Are there any other instances in our codebase 
where we're currently constructing loaders whose behavior we need to 
double-check?


> libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, 
> hdfs_chown, and hdfs_chmod
> 
>
> Key: HDFS-10754
> URL: https://issues.apache.org/jira/browse/HDFS-10754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10754.HDFS-8707.000.patch, 
> HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, 
> HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, 
> HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, 
> HDFS-10754.HDFS-8707.007.patch, HDFS-10754.HDFS-8707.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10340) data node sudden killed

[Kihwal Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432830#comment-15432830
 ] 

Kihwal Lee commented on HDFS-10340:
---

I checked the oom killer code and it is SIGKILL as you pointed out. It might 
have used SIGTERM in the ancient versions. This wouldn't have been caught by 
the sys call snooping, as it does not involve any. It sure looks like something 
else sending SIGTERM to the datanode process. I looked over the openjdk8 source 
but couldn't find anything raising SIGTERM for itself to shutdown.  Whoever the 
sender is, you should be able to catch it with the systemtap instrumentation.

We have had similar issues due to stale pid files, but that can't be it if no 
service was (re)started at that time. 

bq. if user of DataNode is same with NodeManager, maybe it is related with 
YARN-4459
Are you saying that your cluster is configured this way? If so, I agree 
YARN-4459 is a good candidate. If not, we are back to square one.  In any case, 
the systemtap instrumentation should help identifying the source of the signal.

> data node sudden killed 
> 
>
> Key: HDFS-10340
> URL: https://issues.apache.org/jira/browse/HDFS-10340
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Affects Versions: 2.6.0
> Environment: Ubuntu 16.04 LTS , RAM 16g , cpu core : 8 , hdd 100gb, 
> hadoop 2.6.0
>Reporter: tu nguyen khac
>Priority: Critical
>
> I tried to setup a new data node using ubuntu 16 
> and get it join to an existed Hadoop Hdfs cluster ( there are 10 nodes in 
> this cluster and they all run on centos Os 6 ) 
> But when i try to boostrap this node , after about 10 or 20 minutes i get 
> this strange errors : 
> 2016-04-26 20:12:09,394 INFO 
> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: 
> /10.3.24.65:55323, dest: /10.3.24.197:50010, bytes: 79902, op: HDFS_WRITE, 
> cliID: DFSClient_NONMAPREDUCE_1379996362_1, offset: 0, srvID: 
> 225f5b43-1dd3-4ac6-88d2-1e8d27dba55b, blockid: 
> BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, duration: 
> 15331628
> 2016-04-26 20:12:09,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> PacketResponder: BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832, 
> type=LAST_IN_PIPELINE, downstreams=0:[] terminating
> 2016-04-26 20:12:25,410 INFO 
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification 
> succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038502_789829
> 2016-04-26 20:12:25,411 INFO 
> org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification 
> succeeded for BP-352432948-10.3.24.65-1433821675295:blk_1074038505_789832
> 2016-04-26 20:13:18,546 INFO 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
>  Scheduling blk_1074038502_789829 file 
> /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502
>  for deletion
> 2016-04-26 20:13:18,562 INFO 
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
>  Deleted BP-352432948-10.3.24.65-1433821675295 blk_1074038502_789829 file 
> /data/hadoop_data/backup/data/current/BP-352432948-10.3.24.65-1433821675295/current/finalized/subdir4/subdir134/blk_1074038502
> 2016-04-26 20:15:46,481 ERROR 
> org.apache.hadoop.hdfs.server.datanode.DataNode: RECEIVED SIGNAL 15: SIGTERM
> 2016-04-26 20:15:46,504 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: 
> SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down DataNode at bigdata-dw-24-197/10.3.24.197
> /



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Created] (HDFS-10785) libhdfs++: Implement the rest of the tools

[Anatoli Shein (JIRA)]

Anatoli Shein created HDFS-10785:


 Summary: libhdfs++: Implement the rest of the tools
 Key: HDFS-10785
 URL: https://issues.apache.org/jira/browse/HDFS-10785
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Anatoli Shein






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Assigned] (HDFS-10785) libhdfs++: Implement the rest of the tools

[Anatoli Shein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anatoli Shein reassigned HDFS-10785:


Assignee: Anatoli Shein

> libhdfs++: Implement the rest of the tools
> --
>
> Key: HDFS-10785
> URL: https://issues.apache.org/jira/browse/HDFS-10785
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, and hdfs_chmod

[Anatoli Shein (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431644#comment-15431644
 ] 

Anatoli Shein edited comment on HDFS-10754 at 8/23/16 1:48 PM:
---

Somehow recursion counter became not atomic in the last patch. Fixed now.


was (Author: anatoli.shein):
Somehow recursion counter became not atomic in the last patch. Fixed not.

> libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, 
> hdfs_chown, and hdfs_chmod
> 
>
> Key: HDFS-10754
> URL: https://issues.apache.org/jira/browse/HDFS-10754
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10754.HDFS-8707.000.patch, 
> HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, 
> HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, 
> HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, 
> HDFS-10754.HDFS-8707.007.patch, HDFS-10754.HDFS-8707.008.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10679) libhdfs++: Implement parallel find with wildcards tool

[Anatoli Shein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anatoli Shein updated HDFS-10679:
-
Attachment: HDFS-10679.HDFS-8707.013.patch

Separated recursive state, fixed config loader, change constructors to use 
make_shared.

> libhdfs++: Implement parallel find with wildcards tool
> --
>
> Key: HDFS-10679
> URL: https://issues.apache.org/jira/browse/HDFS-10679
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
> Attachments: HDFS-10679.HDFS-8707.000.patch, 
> HDFS-10679.HDFS-8707.001.patch, HDFS-10679.HDFS-8707.002.patch, 
> HDFS-10679.HDFS-8707.003.patch, HDFS-10679.HDFS-8707.004.patch, 
> HDFS-10679.HDFS-8707.005.patch, HDFS-10679.HDFS-8707.006.patch, 
> HDFS-10679.HDFS-8707.007.patch, HDFS-10679.HDFS-8707.008.patch, 
> HDFS-10679.HDFS-8707.009.patch, HDFS-10679.HDFS-8707.010.patch, 
> HDFS-10679.HDFS-8707.011.patch, HDFS-10679.HDFS-8707.012.patch, 
> HDFS-10679.HDFS-8707.013.patch
>
>
> The find tool will issue the GetListing namenode operation on a given 
> directory, and filter the results using posix globbing library.
> If the recursive option is selected, for each returned entry that is a 
> directory the tool will issue another asynchronous call GetListing and repeat 
> the result processing in a recursive fashion.
> One implementation issue that needs to be addressed is the way how results 
> are returned back to the user: we can either buffer the results and return 
> them to the user in bulk, or we can return results continuously as they 
> arrive. While buffering would be an easier solution, returning results as 
> they arrive would be more beneficial to the user in terms of performance, 
> since the result processing can start as soon as the first results arrive 
> without any delay. In order to do that we need the user to use a loop to 
> process arriving results, and we need to send a special message back to the 
> user when the search is over.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10762) Pass IIP for file status related methods

[Daryn Sharp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daryn Sharp updated HDFS-10762:
---
Attachment: HDFS-10762.1.patch

Undid loss of one of my prior changes.  Patch now applies to both branches.

> Pass IIP for file status related methods
> 
>
> Key: HDFS-10762
> URL: https://issues.apache.org/jira/browse/HDFS-10762
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: hdfs
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10762.1.patch, HDFS-10762.patch
>
>
> The frequently called file status methods will not require path re-resolves 
> if the IIP is passed down the call stack.  The code can be simplified further 
> if the IIP tracks if the original path was a reserved raw path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10650) DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory permission

[Xiao Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432759#comment-15432759
 ] 

Xiao Chen commented on HDFS-10650:
--

Thanks Ray for the reminder.

[~jzhuge], could you Edit this jira and input in 'Release Notes' field? Thanks.

> DFSClient#mkdirs and DFSClient#primitiveMkdir should use default directory 
> permission
> -
>
> Key: HDFS-10650
> URL: https://issues.apache.org/jira/browse/HDFS-10650
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10650.001.patch, HDFS-10650.002.patch
>
>
> These 2 DFSClient methods should use default directory permission to create a 
> directory.
> {code:java}
>   public boolean mkdirs(String src, FsPermission permission,
>   boolean createParent) throws IOException {
> if (permission == null) {
>   permission = FsPermission.getDefault();
> }
> {code}
> {code:java}
>   public boolean primitiveMkdir(String src, FsPermission absPermission, 
> boolean createParent)
> throws IOException {
> checkOpen();
> if (absPermission == null) {
>   absPermission = 
> FsPermission.getDefault().applyUMask(dfsClientConf.uMask);
> } 
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10719) In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.

[Karthik Palanisamy (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432716#comment-15432716
 ] 

Karthik Palanisamy commented on HDFS-10719:
---

I simplify the fix here by throwing UnknownHostException instead NullPointer 
exception, also pointing hostname from URI which is unresolved.

*Without fix*

{panel}
2016-08-23 11:54:19,780 ERROR namenode.NameNode (NameNode.java:main(1712)) - 
Failed to start namenode.
java.lang.IllegalArgumentException: Unable to construct journal, 
qjournal://kart2402.xxx.com:8485;kart2401.xxx.com:8485;kart2403.xxx.com:8485/karthik
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1637)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:260)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:789)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:634)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:722)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:951)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:935)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1635)
... 13 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107)
at 
org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91)
at 
org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.(IPCLoggerChannel.java:178)
at 
org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156)
at 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367)
at 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
at 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.(QuorumJournalManager.java:116)
at 
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.(QuorumJournalManager.java:105)
... 18 more
{panel}


*With fix*

{panel}
2016-08-23 12:03:02,621 ERROR namenode.NameNode (NameNode.java:main(1712)) - 
Failed to start namenode.
java.lang.IllegalArgumentException: Unable to construct journal, 
qjournal://kart2402.xxx.com:8485;kart2401.xxx.com:8485;kart2403.xxx.com:8485/karthik
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1637)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282)
at 
org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:260)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:789)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:634)
at 
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:722)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:951)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:935)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
at 

[jira] [Updated] (HDFS-10719) In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.

[Karthik Palanisamy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Palanisamy updated HDFS-10719:
--
Attachment: HDFS-10719-4.patch

> In HA, Namenode is failed to start If any of the Quorum hostname is 
> unresolved.
> ---
>
> Key: HDFS-10719
> URL: https://issues.apache.org/jira/browse/HDFS-10719
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: journal-node, namenode
>Affects Versions: 2.7.1
> Environment: HDP-2.4
>Reporter: Karthik Palanisamy
>Assignee: Karthik Palanisamy
>  Labels: patch
> Attachments: HDFS-10719-1.patch, HDFS-10719-2.patch, 
> HDFS-10719-3.patch, HDFS-10719-4.patch
>
>
> 2016-08-03 02:53:53,760 ERROR namenode.NameNode (NameNode.java:main(1712)) - 
> Failed to start namenode.
> java.lang.IllegalArgumentException: Unable to construct journal, 
> qjournal://1:8485;2:8485;3:8485/shva
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1637)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:260)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:789)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:634)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:951)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:935)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1635)
> ... 13 more
> Caused by: java.lang.NullPointerException
> at 
> org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107)
> at 
> org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91)
> at 
> org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.(IPCLoggerChannel.java:178)
> at 
> org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156)
> at 
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367)
> at 
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
> at 
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.(QuorumJournalManager.java:116)
> at 
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.(QuorumJournalManager.java:105)
> ... 18 more
> 2016-08-03 02:53:53,765 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - 
> Exiting with status 1
> 2016-08-03 02:53:53,768 INFO  namenode.NameNode (LogAdapter.java:info(47)) - 
> SHUTDOWN_MSG:
> *and the failover is not successful*
> I have attached the patch, It allows the Namenode to start if the majority of 
> the Quorums are resolvable.
> throws warning if the quorum is unresolvable.
> throws Unknown host exception if the majority of the journals are 
> unresolvable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432658#comment-15432658
 ] 

Wei-Chiu Chuang commented on HDFS-8986:
---

Committed this to trunk. But the patch has an incompatible change in branch-2 
due to HADOOP-11666. If I understand it correctly, the branch-2 patch should 
remove the output of spaceConsumed. But just to be sure, [~xiaochen] could you 
rebase and make a branch-2 patch for a precommit check?

Thanks.

> Add option to -du to calculate directory space usage excluding snapshots
> 
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch, 
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, 
> HDFS-8986.06.patch, HDFS-8986.07.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its 
> children), the report includes space consumed by blocks that are only present 
> in the snapshots. This is confusing for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related 
> disk usage in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432648#comment-15432648
 ] 

Hudson commented on HDFS-8986:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10327 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10327/])
HDFS-8986. Add option to -du to calculate directory space usage (weichiu: rev 
f0efea490e5aa9dd629d2199aae9c5b1290a17ee)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
* (edit) hadoop-common-project/hadoop-common/src/test/resources/testConf.xml
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/shell/TestCount.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/FsUsage.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Count.java
* (edit) 
hadoop-common-project/hadoop-common/src/site/markdown/FileSystemShell.md
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelperClient.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ContentSummary.java
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/hdfs.proto


> Add option to -du to calculate directory space usage excluding snapshots
> 
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch, 
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, 
> HDFS-8986.06.patch, HDFS-8986.07.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its 
> children), the report includes space consumed by blocks that are only present 
> in the snapshots. This is confusing for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related 
> disk usage in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432629#comment-15432629
 ] 

Hudson commented on HDFS-10783:
---

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10326 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/10326/])
HDFS-10783. The option '-maxSize' and '-step' fail in (aajisaka: rev 
e90f3359de299ef5e3a54ca71070e3dfe1dbb98c)
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewer.java
* (edit) 
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java


> The option '-maxSize' and '-step' fail in OfflineImageViewer
> 
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch, 
> HDFS-10783.003.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the 
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: 
> -maxSize
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-8986) Add option to -du to calculate directory space usage excluding snapshots

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-8986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432621#comment-15432621
 ] 

Wei-Chiu Chuang commented on HDFS-8986:
---

+1 committing this.

> Add option to -du to calculate directory space usage excluding snapshots
> 
>
> Key: HDFS-8986
> URL: https://issues.apache.org/jira/browse/HDFS-8986
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: snapshots
>Reporter: Gautam Gopalakrishnan
>Assignee: Xiao Chen
> Attachments: HDFS-8986.01.patch, HDFS-8986.02.patch, 
> HDFS-8986.03.patch, HDFS-8986.04.patch, HDFS-8986.05.patch, 
> HDFS-8986.06.patch, HDFS-8986.07.patch
>
>
> When running {{hadoop fs -du}} on a snapshotted directory (or one of its 
> children), the report includes space consumed by blocks that are only present 
> in the snapshots. This is confusing for end users.
> {noformat}
> $  hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -createSnapshot /tmp/parent snap1
> Created snapshot /tmp/parent/.snapshot/snap1
> $ hadoop fs -rm -skipTrash /tmp/parent/sub1/*
> ...
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 799.7 M  2.3 G  /tmp/parent
> 799.7 M  2.3 G  /tmp/parent/sub1
> $ hdfs dfs -deleteSnapshot /tmp/parent snap1
> $ hadoop fs -du -h -s /tmp/parent /tmp/parent/*
> 0  0  /tmp/parent
> 0  0  /tmp/parent/sub1
> {noformat}
> It would be helpful if we had a flag, say -X, to exclude any snapshot related 
> disk usage in the output



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Updated] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HDFS-10783:
-
   Resolution: Fixed
 Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
   2.8.0
   Status: Resolved  (was: Patch Available)

Committed this to trunk, branch-2, and branch-2.8. Thanks [~linyiqun] for the 
contribution!

> The option '-maxSize' and '-step' fail in OfflineImageViewer
> 
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Fix For: 2.8.0, 3.0.0-alpha2
>
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch, 
> HDFS-10783.003.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the 
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: 
> -maxSize
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer

[Akira Ajisaka (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432612#comment-15432612
 ] 

Akira Ajisaka commented on HDFS-10783:
--

+1, checking this in.

> The option '-maxSize' and '-step' fail in OfflineImageViewer
> 
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch, 
> HDFS-10783.003.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the 
> following exception:
> {code}
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: 
> -maxSize
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10756) Expose getTrashRoot to HTTPFS and WebHDFS

[Wei-Chiu Chuang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432606#comment-15432606
 ] 

Wei-Chiu Chuang commented on HDFS-10756:


Hello [~yuanbo], nice work!
On top of Xiao's comment, I have a few quick comments:

* when logging an exception, please make sure to log the exception itself. The 
exception itself (including the log message and the stacktrace) is a useful 
information when debugging.
* can you also verify the behavior of getTrashRoot() with a non-EZ path?

Also, I would like to point out that for any downstream applications that use 
the proposed GETTRASHROOT operation, it is important to note that the 
application should not assume the trash directory returned exists. Downstream 
applications should have some way of handling this, and create trash directory 
with appropriate sticky bit and permission. See HDFS-10324 for discussion.

> Expose getTrashRoot to HTTPFS and WebHDFS
> -
>
> Key: HDFS-10756
> URL: https://issues.apache.org/jira/browse/HDFS-10756
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: encryption, httpfs, webhdfs
>Reporter: Xiao Chen
>Assignee: Yuanbo Liu
> Attachments: HDFS-10756.001.patch
>
>
> Currently, hadoop FileSystem API has 
> [getTrashRoot|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L2708]
>  to determine trash directory at run time. Default trash dir is under 
> {{/user/$USER}}
> For an encrypted file, since moving files between/in/out of EZs are not 
> allowed, when an EZ file is deleted via CLI, it calls in to [DFS 
> implementation|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java#L2485]
>  to move the file to a trash directory under the same EZ.
> This works perfectly fine for CLI users or java users who call FileSystem 
> API. But for users via httpfs/webhdfs, currently there is no way to figure 
> out what the trash root would be. This jira is proposing we add such 
> interface to httpfs and webhdfs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-10783) The option '-maxSize' and '-step' fail in OfflineImageViewer

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-10783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432479#comment-15432479
 ] 

Hadoop QA commented on HDFS-10783:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 75m 
19s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12825004/HDFS-10783.003.patch |
| JIRA Issue | HDFS-10783 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux f77bf9cf388b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / c49333b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16507/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: 
hadoop-hdfs-project/hadoop-hdfs |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16507/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> The option '-maxSize' and '-step' fail in OfflineImageViewer
> 
>
> Key: HDFS-10783
> URL: https://issues.apache.org/jira/browse/HDFS-10783
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: tools
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
> Attachments: HDFS-10783.001.patch, HDFS-10783.002.patch, 
> HDFS-10783.003.patch
>
>
> Executing -step or -maxSize option in {{OfflineImageViewer}} will cause the 
> following exception:
> {code}
> 

[jira] [Commented] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432446#comment-15432446
 ] 

Hadoop QA commented on HDFS-9785:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 81 unchanged - 3 fixed = 81 total (was 84) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
52s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12787431/HDFS-9785.001.patch |
| JIRA Issue | HDFS-9785 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a5c4e732a786 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8cc4a67 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16509/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16509/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-9785.001.patch
>
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs 
> with Path is not used anymore.



--

[jira] [Commented] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432434#comment-15432434
 ] 

Hadoop QA commented on HDFS-9785:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-common-project/hadoop-common: The patch 
generated 0 new + 81 unchanged - 3 fixed = 81 total (was 84) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12787431/HDFS-9785.001.patch |
| JIRA Issue | HDFS-9785 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 6f9e9eec034f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8cc4a67 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16508/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U: 
hadoop-common-project/hadoop-common |
| Console output | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16508/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-9785.001.patch
>
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs 
> with Path is not used 

[jira] [Updated] (HDFS-9785) Remove unused TrashPolicy#getInstance and initiate code

[Akira Ajisaka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-9785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka updated HDFS-9785:

Target Version/s:   (was: )
  Labels:   (was: incompatibleChange)
Hadoop Flags: Incompatible change

> Remove unused TrashPolicy#getInstance and initiate code
> ---
>
> Key: HDFS-9785
> URL: https://issues.apache.org/jira/browse/HDFS-9785
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-9785.001.patch
>
>
> A follow-on from HDFS-8831: now the {{getInstance}} and {{initiate}} APIs 
> with Path is not used anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-9392) Admins support for maintenance state

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432321#comment-15432321
 ] 

Hadoop QA commented on HDFS-9392:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 7 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 31s{color} | {color:orange} hadoop-hdfs-project: The patch generated 12 new 
+ 385 unchanged - 28 fixed = 397 total (was 413) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
52s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 37s{color} 
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 33s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks 
|
|   | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:9560f25 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12824992/HDFS-9392-3.patch |
| JIRA Issue | HDFS-9392 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a014b3c0f914 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / c49333b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16506/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-HDFS-Build/16506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
 |
|  Test Results |