[jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5776: Attachment: HDFS-5776-v8.txt > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt, > HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880783#comment-13880783 ] Liang Xie commented on HDFS-5776: - v8 add the enoughNodesForHedgedRead() function to sanity check, [~saint@gmail.com]'s comments is great, we definitely need a switch per DFSClient instance. bq. the number of threads config needs to be NumberOfHBaseOpenFiles X 2 else the feature will not work for all files still works, but probably lots of requests will execute in current thread, that means no latency benefit from hedged read feature. this's is a good requirement that we need a per client instance's switch, such that we can let some instances use this feature, we can control it on demand, right ? :) > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776-v8.txt, > HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880762#comment-13880762 ] Liang Xie commented on HDFS-5776: - bq. Can this scenario be possible? In hedgedFetchBlockByteRange, if we hit the timeout for the first DN, we will add the DN to the ignore list, and call chooseDataNode again. If the first DN is the only DN we can read, we will get IOException from bestNode. Then we will run into a loop where we keep trying to get another DN multiple times (some NN rpc call will even be fired). And during this process the first DN can even return the data. In this scenario I guess we may get a worse performance? Thus I guess we should not trigger hedged read if we find that we cannot (easily) find the second DN for read? yes,there's possible happen about your case, nice! and a very easy handling method is just introduce a double-check function, say enoughNodesForHedgedRead(LocatedBlock block) into the pread checking code branch > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880757#comment-13880757 ] stack commented on HDFS-5776: - [~xieliang007] what you think of the new comments above by the lads? Now the executor is static, the number of threads config needs to be NumberOfHBaseOpenFiles X 2 else the feature will not work for all files? Thanks. > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5776: Attachment: HDFS-5776-v7.txt Attached v7 makes the pool static now, please review > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776-v7.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880746#comment-13880746 ] stack commented on HDFS-5776: - bq. An alternative way to do this is to have an "Allow-Hedged-Reads" configuration, and if it is set to true, we load the number of thread pool and the threshold time. We will provide an isHedgedReadsEnabled method but we will not provide enable/disable methods. The reviews are great. On the above, while I can see putting the on/off switch as a DN config., we should allow setting at least the config on when to start the hedge read per DFSCient instance. bq. This means that we do need to put this in a static, for now, or else FileContext users will be constantly destroying and creating thread-pools. Thanks Colin. Makes sense. > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
[ https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880721#comment-13880721 ] Vinay commented on HDFS-5723: - bq. Hi Vinay, one question about the patch: so this inconsistent generation stamp can also be caused by a delayed block-received report? I.e., after the first close(), the DN's report gets delayed and is received by NN when the append starts. In that case, will we have any issue by wrongly putting the (block, DN) into the corruptBlockMap? It can happen. Earlier block with prev genstamp will be marked corrupt, if append pipeline setup with genstamp update happened before previous report comes. But append pipeline creation will update the genstamp in that datanode also, so one more block-received report is expectedn with correct genstamp. This time it will remove the old block from corrupt replica map. > Append failed FINALIZED replica should not be accepted as valid when that > block is underconstruction > > > Key: HDFS-5723 > URL: https://issues.apache.org/jira/browse/HDFS-5723 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5723.patch > > > Scenario: > 1. 3 node cluster with > dfs.client.block.write.replace-datanode-on-failure.enable set to false. > 2. One file is written with 3 replicas, blk_id_gs1 > 3. One of the datanode DN1 is down. > 4. File was opened with append and some more data is added to the file and > synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2 > 5. Now DN1 restarted > 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should > be marked corrupted. > but since NN having appended block state as UnderConstruction, at this time > its not detecting this block as corrupt and adding to valid block locations. > As long as the namenode is alive, this datanode also will be considered as > valid replica and read/append will fail in that datanode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5809) BlockPoolSliceScanner make datanode to drop into infinite loop
[ https://issues.apache.org/jira/browse/HDFS-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880702#comment-13880702 ] ikweesung commented on HDFS-5809: - My ugly path like this: {code} if ( info != null ) { delBlockInfo(info); } else if(blockInfoSet.contains(block)){ delBlockInfo(info); info = new BlockScanInfo(block); } else { // It might already be removed. Thats ok, it will be caught next time. info = new BlockScanInfo(block); } {code} > BlockPoolSliceScanner make datanode to drop into infinite loop > -- > > Key: HDFS-5809 > URL: https://issues.apache.org/jira/browse/HDFS-5809 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.0.0-alpha > Environment: jdk1.6, centos6.4, 2.0.0-cdh4.5.0 >Reporter: ikweesung >Priority: Critical > Labels: blockpoolslicescanner, datanode, infinite-loop > > Hello, everyone. > When hadoop cluster starts, BlockPoolSliceScanner start scanning the blocks > in my cluster. > Then, randomly one datanode drop into infinite loop as the log show, and > finally all datanodes drop into infinite loop. > Every datanode just verify fail by one block. > When i check the fail block like this : hadoop fsck / -files -blocks | grep > blk_1223474551535936089_4702249, no hdfs file contains the block. > It seems that in while block of BlockPoolSliceScanner's scan method drop into > infinite loop . > BlockPoolSliceScanner: 650 > while (datanode.shouldRun > && !datanode.blockScanner.blockScannerThread.isInterrupted() > && datanode.isBPServiceAlive(blockPoolId)) { > The log finally printed in method verifyBlock(BlockPoolSliceScanner:453). > Please excuse my poor English. > - > LOG: > 2014-01-21 18:36:50,582 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > failed for > BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634 - > may be due to race with write > 2014-01-21 18:36:50,582 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > failed for > BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634 - > may be due to race with write > 2014-01-21 18:36:50,582 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > failed for > BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634 - > may be due to race with write -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5809) BlockPoolSliceScanner make datanode to drop into infinite loop
[ https://issues.apache.org/jira/browse/HDFS-5809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880700#comment-13880700 ] ikweesung commented on HDFS-5809: - Please execute my poor English. : ) I found that int BlockPoolSliceScanner, blockInfoSet can contain two block which has the same block id, because BlockScanInfo compare by lastScanTime. Then int method updateScanStatus, the BlockScanInfo can not be updated, so ((now - getEarliestScanTime()) >= scanPeriod) will be always true. This cause datanode drop into infinite loop. > BlockPoolSliceScanner make datanode to drop into infinite loop > -- > > Key: HDFS-5809 > URL: https://issues.apache.org/jira/browse/HDFS-5809 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.0.0-alpha > Environment: jdk1.6, centos6.4, 2.0.0-cdh4.5.0 >Reporter: ikweesung >Priority: Critical > Labels: blockpoolslicescanner, datanode, infinite-loop > > Hello, everyone. > When hadoop cluster starts, BlockPoolSliceScanner start scanning the blocks > in my cluster. > Then, randomly one datanode drop into infinite loop as the log show, and > finally all datanodes drop into infinite loop. > Every datanode just verify fail by one block. > When i check the fail block like this : hadoop fsck / -files -blocks | grep > blk_1223474551535936089_4702249, no hdfs file contains the block. > It seems that in while block of BlockPoolSliceScanner's scan method drop into > infinite loop . > BlockPoolSliceScanner: 650 > while (datanode.shouldRun > && !datanode.blockScanner.blockScannerThread.isInterrupted() > && datanode.isBPServiceAlive(blockPoolId)) { > The log finally printed in method verifyBlock(BlockPoolSliceScanner:453). > Please excuse my poor English. > - > LOG: > 2014-01-21 18:36:50,582 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > failed for > BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634 - > may be due to race with write > 2014-01-21 18:36:50,582 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > failed for > BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634 - > may be due to race with write > 2014-01-21 18:36:50,582 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > failed for > BP-1040548460-58.229.158.13-1385606058039:blk_6833233229840997944_4702634 - > may be due to race with write -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5318) Support read-only and read-write paths to shared replicas
[ https://issues.apache.org/jira/browse/HDFS-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880680#comment-13880680 ] Arpit Agarwal commented on HDFS-5318: - Hi Eric, I'm going to think about this some more. My first thought is that your plugin implementation is doing the right thing and there is no good reason for reporting {{READ_ONLY_SHARED}} replicas to NN unless finalized. However if not done right this could result in reported blocks being 'lost' and as you mentioned, skipping ROS storages in {{BlockInfoUnderConstruction.addReplicaIfNotPresent}} is a bad idea. I'm sure you are following HDFS-5194. FsDatasetSpi implementers are required to know too much 'out-of-band' knowledge of about the NN-DN protocol. Ideally the interface could be redone to avoid these problems. For now it may be an acceptable solution to just document this. > Support read-only and read-write paths to shared replicas > - > > Key: HDFS-5318 > URL: https://issues.apache.org/jira/browse/HDFS-5318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Eric Sirianni > Attachments: HDFS-5318.patch, HDFS-5318a-branch-2.patch, > HDFS-5318b-branch-2.patch, HDFS-5318c-branch-2.patch, hdfs-5318.pdf > > > There are several use cases for using shared-storage for datanode block > storage in an HDFS environment (storing cold blocks on a NAS device, Amazon > S3, etc.). > With shared-storage, there is a distinction between: > # a distinct physical copy of a block > # an access-path to that block via a datanode. > A single 'replication count' metric cannot accurately capture both aspects. > However, for most of the current uses of 'replication count' in the Namenode, > the "number of physical copies" aspect seems to be the appropriate semantic. > I propose altering the replication counting algorithm in the Namenode to > accurately infer distinct physical copies in a shared storage environment. > With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor > additional semantics to the {{StorageID}} - namely that multiple datanodes > attaching to the same physical shared storage pool should report the same > {{StorageID}} for that pool. A minor modification would be required in the > DataNode to enable the generation of {{StorageID}} s to be pluggable behind > the {{FsDatasetSpi}} interface. > With those semantics in place, the number of physical copies of a block in a > shared storage environment can be calculated as the number of _distinct_ > {{StorageID}} s associated with that block. > Consider the following combinations for two {{(DataNode ID, Storage ID)}} > pairs {{(DN_A, S_A) (DN_B, S_B)}} for a given block B: > * {{DN_A != DN_B && S_A != S_B}} - *different* access paths to *different* > physical replicas (i.e. the traditional HDFS case with local disks) > ** → Block B has {{ReplicationCount == 2}} > * {{DN_A != DN_B && S_A == S_B}} - *different* access paths to the *same* > physical replica (e.g. HDFS datanodes mounting the same NAS share) > ** → Block B has {{ReplicationCount == 1}} > For example, if block B has the following location tuples: > * {{DN_1, STORAGE_A}} > * {{DN_2, STORAGE_A}} > * {{DN_3, STORAGE_B}} > * {{DN_4, STORAGE_B}}, > the effect of this proposed change would be to calculate the replication > factor in the namenode as *2* instead of *4*. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5797) Implement offline image viewer.
[ https://issues.apache.org/jira/browse/HDFS-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5797: - Attachment: HDFS-5797.001.patch Fix an incorrect format string in LsrPBImage > Implement offline image viewer. > --- > > Key: HDFS-5797 > URL: https://issues.apache.org/jira/browse/HDFS-5797 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5797.000.patch, HDFS-5797.001.patch > > > The format of FSImage has changed dramatically therefore a new implementation > of OfflineImageViewer is required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5825) Use FileUtils.copyFile() to implement DFSTestUtils.copyFile()
[ https://issues.apache.org/jira/browse/HDFS-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880584#comment-13880584 ] Arpit Agarwal commented on HDFS-5825: - +1 (pending Jenkins). > Use FileUtils.copyFile() to implement DFSTestUtils.copyFile() > - > > Key: HDFS-5825 > URL: https://issues.apache.org/jira/browse/HDFS-5825 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5825.000.patch > > > {{DFSTestUtils.copyFile()}} is implemented by copying data through > FileInputStream / FileOutputStream. Apache Common IO provides > {{FileUtils.copyFile()}}. It uses FileChannel which is more efficient. > This jira proposes to implement {{DFSTestUtils.copyFile()}} using > {{FileUtils.copyFile()}}. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5086) Support RPCSEC_GSS authentication in NFSv3 gateway
[ https://issues.apache.org/jira/browse/HDFS-5086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5086: Attachment: HDFS-5086.000.patch An in-progress patch. > Support RPCSEC_GSS authentication in NFSv3 gateway > -- > > Key: HDFS-5086 > URL: https://issues.apache.org/jira/browse/HDFS-5086 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Jing Zhao > Attachments: HDFS-5086.000.patch > > -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880577#comment-13880577 ] Abin Shahab commented on HDFS-5804: --- BTW, I have a patch that gets rid off even checking whether we are in secure mode, but I'm not sure if it's the right thing to submit that patch. That patch would require the nfs-gateway user(nfsserver in our case) be allowed to proxy root, even in non-secure mode. That's a big change. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.channel.SimpleCh
[jira] [Updated] (HDFS-5826) Update the stored edit logs to be consistent with the changes in HDFS-5698 branch
[ https://issues.apache.org/jira/browse/HDFS-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5826: - Attachment: HDFS-5826.000.patch > Update the stored edit logs to be consistent with the changes in HDFS-5698 > branch > - > > Key: HDFS-5826 > URL: https://issues.apache.org/jira/browse/HDFS-5826 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: HDFS-5698 (FSImage in protobuf) >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-5826.000.patch > > > HDFS-5698 bumps the LayoutVersion to indicate whether the file is in new > format. The stored edit logs have to be updated in order to pass > {{testOfflineEditsViewer}}. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880572#comment-13880572 ] Jing Zhao commented on HDFS-5804: - Sure, I will post what I have to HDFS-5086. In general, I was just trying to merge the GSS authentication part from [~brocknoland]'s NFS4 implementation (https://github.com/cloudera/hdfs-nfs-proxy) into the current NFS3-based implementation. You can directly check [~brocknoland]'s implementation also. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.chan
[jira] [Created] (HDFS-5826) Update the stored edit logs to be consistent with the changes in HDFS-5698 branch
Haohui Mai created HDFS-5826: Summary: Update the stored edit logs to be consistent with the changes in HDFS-5698 branch Key: HDFS-5826 URL: https://issues.apache.org/jira/browse/HDFS-5826 Project: Hadoop HDFS Issue Type: Bug Affects Versions: HDFS-5698 (FSImage in protobuf) Reporter: Haohui Mai Assignee: Haohui Mai HDFS-5698 bumps the LayoutVersion to indicate whether the file is in new format. The stored edit logs have to be updated in order to pass {{testOfflineEditsViewer}}. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5608) WebHDFS: implement GETACLSTATUS and SETACL.
[ https://issues.apache.org/jira/browse/HDFS-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880565#comment-13880565 ] Chris Nauroth commented on HDFS-5608: - Comments on the latest version of the patch: # {{JsonUtil}}: It looks like we never would call the new {{toJsonString}} or {{toAclStatus}} methods with {{includesType}} set to false. Can we remove the {{includesType}} parameter and the code paths that would have handled the {{false}} case? That way, we won't have dormant, untested code. # {{JsonUtil#toAclStatus}}: Inside the for loop, we're still repeating parsing logic that also exists in {{AclEntry#parseAclSpec}}. I think we need one more refactoring in the common code to give us a static {{AclEntry#parseAclEntry}} method that parses a single ACL entry (not a list like {{AclEntry#parseAclSpec}}). I'll let you know when that's available. # {{AclPermissionParam}}: The constructor makes 3 separate calls to {{parseAclSpec}}. That method cannot return null, so you can cut down to 2 calls by removing the null check. # {{AclPermissionParam#parseAclSpec}}: There is a typo in the variable name {{aclspce}}. Inside the for loop, you can use {{AclEntry#toString}} instead of building the string yourself. This code has some bugs like not prepending "default" for a default ACL and a {{NullPointerException}} getting the permission symbol if no permission is defined (as {{removeAclEntries}} does). You'll get them all fixed for free if you switch to {{AclEntry#toString}}. Actually, it seems like this whole method is equivalent to this one-liner: {{StringUtils.join(",", entries)}}. Try that out. # {{TestJsonUtil}}: Thanks for adding tests! I recently added a class named {{AclTestHelpers}} that contains some static methods that can shorten test code that needs to make ACL entries. You can see an example of how this is used in places like {{TestNameNodeAcl}}. # In addition to the tests already in the patch, we'll also need to add functional tests that exercise each of the new APIs end-to-end using a {{MiniDFSCluster}}. Let's add these tests in {{org.apache.hadoop.hdfs.web.TestWebHDFS}}. I think we'll be ready to commit after the above is addressed, so we're getting close. :-) Thank you for incorporating the feedback. > WebHDFS: implement GETACLSTATUS and SETACL. > --- > > Key: HDFS-5608 > URL: https://issues.apache.org/jira/browse/HDFS-5608 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: webhdfs >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Sachin Jose > Attachments: HDFS-5608.0.patch, HDFS-5608.1.patch, HDFS-5608.2.patch, > HDFS-5608.3.patch > > > Implement and test {{GETACLS}} and {{SETACL}} in WebHDFS. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880550#comment-13880550 ] Abin Shahab commented on HDFS-5804: --- May I take a look at your patch? I was planning to mimic how org.apache.hadoop.ipc.Client does the authentication. Also, I don't have access to assign issues to myself. I would definitely like to assign this one to me. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java
[jira] [Updated] (HDFS-5797) Implement offline image viewer.
[ https://issues.apache.org/jira/browse/HDFS-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5797: - Attachment: HDFS-5797.000.patch > Implement offline image viewer. > --- > > Key: HDFS-5797 > URL: https://issues.apache.org/jira/browse/HDFS-5797 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5797.000.patch > > > The format of FSImage has changed dramatically therefore a new implementation > of OfflineImageViewer is required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5723) Append failed FINALIZED replica should not be accepted as valid when that block is underconstruction
[ https://issues.apache.org/jira/browse/HDFS-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880525#comment-13880525 ] Jing Zhao commented on HDFS-5723: - Hi Vinay, one question about the patch: so this inconsistent generation stamp can also be caused by a delayed block-received report? I.e., after the first close(), the DN's report gets delayed and is received by NN when the append starts. In that case, will we have any issue by wrongly putting the (block, DN) into the corruptBlockMap? > Append failed FINALIZED replica should not be accepted as valid when that > block is underconstruction > > > Key: HDFS-5723 > URL: https://issues.apache.org/jira/browse/HDFS-5723 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Vinay >Assignee: Vinay > Attachments: HDFS-5723.patch > > > Scenario: > 1. 3 node cluster with > dfs.client.block.write.replace-datanode-on-failure.enable set to false. > 2. One file is written with 3 replicas, blk_id_gs1 > 3. One of the datanode DN1 is down. > 4. File was opened with append and some more data is added to the file and > synced. (to only 2 live nodes DN2 and DN3)-- blk_id_gs2 > 5. Now DN1 restarted > 6. In this block report, DN1 reported FINALIZED block blk_id_gs1, this should > be marked corrupted. > but since NN having appended block state as UnderConstruction, at this time > its not detecting this block as corrupt and adding to valid block locations. > As long as the namenode is alive, this datanode also will be considered as > valid replica and read/append will fail in that datanode. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Assigned] (HDFS-5797) Implement offline image viewer.
[ https://issues.apache.org/jira/browse/HDFS-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5797: Assignee: Haohui Mai > Implement offline image viewer. > --- > > Key: HDFS-5797 > URL: https://issues.apache.org/jira/browse/HDFS-5797 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5797.000.patch > > > The format of FSImage has changed dramatically therefore a new implementation > of OfflineImageViewer is required. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880508#comment-13880508 ] Jing Zhao commented on HDFS-5804: - bq. this still allows any user in the proxied group to authenticate WITHOUT having a kerberos ticket. Yeah, currently nfs-gateway can only do simple AUTH_UNIX authentication, thus we need to finish HDFS-5086 so that nfs-gateway can authenticate clients based on kerberos. I have an in-progress patch long time ago, I will see if I can finish it recently. Also feel free to assign that jira to yourself if you want to work on it. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) >
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880491#comment-13880491 ] Abin Shahab commented on HDFS-5804: --- Ah! I see your point. I think I can allow nfsserver to proxy root, and that'd allow this patch to work properly(I've removed the root check condition). BTW, this still allows any user in the proxied group to authenticate WITHOUT having a kerberos ticket. Do you have any advice on implementing the kerberos authentication on the nfs-gateway? We are kerberizing our clusters, and seems like nfs is allowing them to circumvent kerberos authentication. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameD
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880459#comment-13880459 ] Jing Zhao commented on HDFS-5804: - Abin, I see your issue now. So from the nfs-gateway point of view, I think it should just simply impersonate any user who has passed its own authentication, thus should not have special case on root. In HDFS, why do you want to disable the proxy setting for root? HDFS does not respect root as a special user. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.net
[jira] [Commented] (HDFS-4685) Implementation of ACLs in HDFS
[ https://issues.apache.org/jira/browse/HDFS-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880440#comment-13880440 ] Sanjay Radia commented on HDFS-4685: Comment on the the two alternatives for the default ACL proposals in the doc. Reproducing the text for convenience. * *Umask-Default-ACL*: The default ACL of the parent is cloned to the ACL of the child at time of child creation. For new child directories, the default ACL itself is also cloned, so that the same policy is applied to sub-directories of sub-directories. Subsequent changes to the parent’s default ACL will set a different ACL for new children, but will not alter existing children. This matches POSIX behavior. If the administrator wants to change policy on the sub-tree later, then this is performed by inserting a new more restrictive ACL entry at the appropriate sub-tree root (see UC6) and may also need to run a recursive ACL modification (analogous to chmod -R) since existing children are not effected by the new ACL. * *Inherited-Default-ACL*: A child that does not have an ACL of its own inherits its ACL from the nearest ancestor that has defined a default ACL. A child node that requires a different ACL can override the default (like the Umask-Default-ACL). Subsequent changes to the ancestor’s default ACL will cause all children that do not have an ACL to inherit the new ACL regardless of child creation time (unlike Umask-Default-ACL). This model, like the ABAC ACLs (use case UC8), encourages the user to create fewer ACLs (typically on the root of specific subtrees) while the Posix-compliant Umask-Default-ACL is expected to results in larger number of ACLs in the system. It would also make a memory efficient implementation trivial. Note that this model is a deviation from POSIX behavior. Consider the following three sub use cases here 4a) OpenUP child for wide access than the default. 4b) Restrict a child for narrower access than the default. 4c) Change the defaultAcl because you made a mistake originally. Both models support use case 4a and 4b with equal ease. However, with the Inherited-Default-ACL, it is easy to identify children that have overridden the default-ACL - the existence of an ACL means that the user intended to override the default. Also 4c is a natural fit for Inherited-Default-ACL. For the UMask-Default-ACL, every child has an ACL and hence you have to walk down the subtree and compare the ACL with the default to see if the user had intended to override it. I think the Inherited-Default-ACL is much better design but posix compliance may triumph and hence am willing to go with UMask-Default-ACL. > Implementation of ACLs in HDFS > -- > > Key: HDFS-4685 > URL: https://issues.apache.org/jira/browse/HDFS-4685 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs-client, namenode, security >Affects Versions: 1.1.2 >Reporter: Sachin Jose >Assignee: Chris Nauroth > Attachments: HDFS-ACLs-Design-1.pdf, HDFS-ACLs-Design-2.pdf > > > Currenly hdfs doesn't support Extended file ACL. In unix extended ACL can be > achieved using getfacl and setfacl utilities. Is there anybody working on > this feature ? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5825) Use FileUtils.copyFile() to implement DFSTestUtils.copyFile()
[ https://issues.apache.org/jira/browse/HDFS-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5825: - Attachment: HDFS-5825.000.patch > Use FileUtils.copyFile() to implement DFSTestUtils.copyFile() > - > > Key: HDFS-5825 > URL: https://issues.apache.org/jira/browse/HDFS-5825 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5825.000.patch > > > {{DFSTestUtils.copyFile()}} is implemented by copying data through > FileInputStream / FileOutputStream. Apache Common IO provides > {{FileUtils.copyFile()}}. It uses FileChannel which is more efficient. > This jira proposes to implement {{DFSTestUtils.copyFile()}} using > {{FileUtils.copyFile()}}. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5825) Use FileUtils.copyFile() to implement DFSTestUtils.copyFile()
[ https://issues.apache.org/jira/browse/HDFS-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5825: - Status: Patch Available (was: Open) > Use FileUtils.copyFile() to implement DFSTestUtils.copyFile() > - > > Key: HDFS-5825 > URL: https://issues.apache.org/jira/browse/HDFS-5825 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5825.000.patch > > > {{DFSTestUtils.copyFile()}} is implemented by copying data through > FileInputStream / FileOutputStream. Apache Common IO provides > {{FileUtils.copyFile()}}. It uses FileChannel which is more efficient. > This jira proposes to implement {{DFSTestUtils.copyFile()}} using > {{FileUtils.copyFile()}}. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Moved] (HDFS-5825) Use FileUtils.copyFile() to implement DFSTestUtils.copyFile()
[ https://issues.apache.org/jira/browse/HDFS-5825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai moved HADOOP-10271 to HDFS-5825: --- Key: HDFS-5825 (was: HADOOP-10271) Project: Hadoop HDFS (was: Hadoop Common) > Use FileUtils.copyFile() to implement DFSTestUtils.copyFile() > - > > Key: HDFS-5825 > URL: https://issues.apache.org/jira/browse/HDFS-5825 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai >Priority: Minor > Attachments: HDFS-5825.000.patch > > > {{DFSTestUtils.copyFile()}} is implemented by copying data through > FileInputStream / FileOutputStream. Apache Common IO provides > {{FileUtils.copyFile()}}. It uses FileChannel which is more efficient. > This jira proposes to implement {{DFSTestUtils.copyFile()}} using > {{FileUtils.copyFile()}}. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880395#comment-13880395 ] Abin Shahab commented on HDFS-5804: --- Jing, Thanks a lot for looking at the issue. I think you've captured what I'm trying to do very well! Thanks for that. Yes. We specifically do not want nfsserver(the user running the nfs-gateway) to be able to impersonate root. We need root for one thing, and only one thing: to mount the filesystem. After that, root is irrelevant, and should not have any access to do anything. Regretably, it does an FSINFO as part of the mount. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880379#comment-13880379 ] Jing Zhao commented on HDFS-5804: - So I guess that idea here is that the nfs gateway acts a service, and authenticates itself through Kerberos to Hadoop/HDFS. Then for the clients of nfs, if a client can authenticate itself in the NFS gateway (currently we only support AUTH_UNIX, and we plan to support GSS in HDFS-5539), the nfs gateway will create a proxy user for the client and use the proxy user to communicate with HDFS. Back to the exception, I have not tested myself, but have you add the proxy user setting in your HDFS's configuration? Because I saw the exception msg is "User: nfsserver/krb-nfs-desktop.my.company@krb.altiscale.com is not allowed to impersonate root". > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:
[jira] [Updated] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abin Shahab updated HDFS-5804: -- Attachment: exception-as-root.log This is the exception I get now. ROOT is doing the mount of nfs. As part of the mount, it issues an FSINFO call, which fails, and it fails the mount. I propose we catch and log the Access control exception for this failure, but not necessary fail the mount. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, exception-as-root.log, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChan
[jira] [Updated] (HDFS-5808) Implement cancellation when saving FSImage
[ https://issues.apache.org/jira/browse/HDFS-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5808: - Description: The code should be able to cancel the progress of saving fsimage / checkpointing. When fail over happens, the code needs to timely cancel the checkpoint operations so that the fail over sequences can proceed. The same functionality exists in the old code. This jira proposes to implement the same functionality in the new fsimage code. was:This jira proposes to implement checking whether the user has cancel the operation when saving the fsimage. > Implement cancellation when saving FSImage > -- > > Key: HDFS-5808 > URL: https://issues.apache.org/jira/browse/HDFS-5808 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5808.000.patch, HDFS-5808.001.patch > > > The code should be able to cancel the progress of saving fsimage / > checkpointing. When fail over happens, the code needs to timely cancel the > checkpoint operations so that the fail over sequences can proceed. > The same functionality exists in the old code. This jira proposes to > implement the same functionality in the new fsimage code. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5808) Implement cancellation when saving FSImage
[ https://issues.apache.org/jira/browse/HDFS-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880296#comment-13880296 ] Suresh Srinivas commented on HDFS-5808: --- Can you please add description on why this is needed (use case) and what the patch does? > Implement cancellation when saving FSImage > -- > > Key: HDFS-5808 > URL: https://issues.apache.org/jira/browse/HDFS-5808 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5808.000.patch, HDFS-5808.001.patch > > > This jira proposes to implement checking whether the user has cancel the > operation when saving the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abin Shahab updated HDFS-5804: -- Attachment: HDFS-5804.patch Updated patch does not have special case on root. Tested with nfs-gateway running as a non-root kerberized user. > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, HDFS-5804.patch, > javadoc-after-patch.log, javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.messageReceived(RpcUtil.java:132) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) > at > org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.Defau
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880284#comment-13880284 ] Colin Patrick McCabe commented on HDFS-5776: By the way, my previous comment was assuming that the alternative proposed to making the thread-pool static was putting it in DFSClient (not a good option). Another option would be making the thread-pool local to the DFSInputStream. However, this seems like it will tend to create an enormous number of threads, especially for applications like HBase that open many files. So again I would argue it should be static. > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880280#comment-13880280 ] Colin Patrick McCabe commented on HDFS-5776: bq. One note on 1.. Is static ever a good idea for sharing resources? But your point of being able to share amongst DFSClient instances is for sure something we should pursue (in another JIRA?) Unfortunately, the {{FileContext}} API creates a new {{DFSClient}} instance for each operation that it does. (The older {{FileSystem}} API doesn't have this problem, since the {{DistributedFileSystem}} object hangs on to the {{DFSClient}} for a while.) This means that we do need to put this in a static, for now, or else {{FileContext}} users will be constantly destroying and creating thread-pools. I have another change pending which creates the concept of a "cache context," where different threads can use different contexts if they like. For now, let's use a static variable, maybe with a TODO. bq. Related to the previous - what do you think of not exposing the DFS_DFSCLIENT_HEDGED_READ_THREADPOOL_SIZE setting at all? Maybe we can just expose a boolean setting to enable it. The reason I prefer not to surface such settings is because it invites abuse (the concern is not with trusted apps like HBase). If we do expose this setting we should at least have an internal upper bound. I don't see why we wouldn't expose this setting. It doesn't give the client the ability to do anything bad it couldn't already do. You can already try to open a zillion files at once in order to attack the {{NameNode}} / {{DataNodes}}. Preventing denial-of-service attacks is not currently something we try to do. And in the future, if we ever do try to prevent denial-of-service attacks, I don't think having hedged reads makes that any more or less difficult than it would otherwise be. > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880279#comment-13880279 ] Jing Zhao commented on HDFS-5776: - # In DFSClient, I agree with Arpit that we should remove the allowHedgedReads field and the enable/disable methods. In the current code, whether hedged read is enabled is determined by the initial setting of the hedgedReadThreadPool. If we provide these extra enable/disable methods, what if a user of DFSClient sets 0 to the thread pool size and later call the enableHedgedReads? Unless we have a clear use case to support the usage of the enable/disable methods, I guess we do not need to provide these flexibility here. An alternative way to do this is to have an "Allow-Hedged-Reads" configuration, and if it is set to true, we load the number of thread pool and the threshold time. We will provide an isHedgedReadsEnabled method but we will not provide enable/disable methods. I guess this may be easier for users to understand. # Can this scenario be possible? In hedgedFetchBlockByteRange, if we hit the timeout for the first DN, we will add the DN to the ignore list, and call chooseDataNode again. If the first DN is the only DN we can read, we will get IOException from bestNode. Then we will run into a loop where we keep trying to get another DN multiple times (some NN rpc call will even be fired). And during this process the first DN can even return the data. In this scenario I guess we may get a worse performance? Thus I guess we should not trigger hedged read if we find that we cannot (easily) find the second DN for read? > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5808) Implement cancellation when saving FSImage
[ https://issues.apache.org/jira/browse/HDFS-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-5808. - Resolution: Fixed Hadoop Flags: Reviewed +1. I've committed this. > Implement cancellation when saving FSImage > -- > > Key: HDFS-5808 > URL: https://issues.apache.org/jira/browse/HDFS-5808 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5808.000.patch, HDFS-5808.001.patch > > > This jira proposes to implement checking whether the user has cancel the > operation when saving the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5808) Implement cancellation when saving FSImage
[ https://issues.apache.org/jira/browse/HDFS-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5808: - Attachment: HDFS-5808.001.patch Rebased > Implement cancellation when saving FSImage > -- > > Key: HDFS-5808 > URL: https://issues.apache.org/jira/browse/HDFS-5808 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Haohui Mai >Assignee: Haohui Mai > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5808.000.patch, HDFS-5808.001.patch > > > This jira proposes to implement checking whether the user has cancel the > operation when saving the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5824) Add a Type field in Snapshot DiffEntry's protobuf definition
[ https://issues.apache.org/jira/browse/HDFS-5824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-5824. - Resolution: Fixed Fix Version/s: HDFS-5698 (FSImage in protobuf) Hadoop Flags: Reviewed Thanks for the review, Haohui! I've committed this. > Add a Type field in Snapshot DiffEntry's protobuf definition > > > Key: HDFS-5824 > URL: https://issues.apache.org/jira/browse/HDFS-5824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Fix For: HDFS-5698 (FSImage in protobuf) > > Attachments: HDFS-5824.000.patch > > > We need to add a Type field to differentiate FileDiff and DirectoryDiff in > our protobuf to enable the offline image viewer to parse the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5824) Add a Type field in Snapshot DiffEntry's protobuf definition
[ https://issues.apache.org/jira/browse/HDFS-5824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880227#comment-13880227 ] Haohui Mai commented on HDFS-5824: -- +1 > Add a Type field in Snapshot DiffEntry's protobuf definition > > > Key: HDFS-5824 > URL: https://issues.apache.org/jira/browse/HDFS-5824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-5824.000.patch > > > We need to add a Type field to differentiate FileDiff and DirectoryDiff in > our protobuf to enable the offline image viewer to parse the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5824) Add a Type field in Snapshot DiffEntry's protobuf definition
[ https://issues.apache.org/jira/browse/HDFS-5824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5824: Attachment: HDFS-5824.000.patch > Add a Type field in Snapshot DiffEntry's protobuf definition > > > Key: HDFS-5824 > URL: https://issues.apache.org/jira/browse/HDFS-5824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-5824.000.patch > > > We need to add a Type field to differentiate FileDiff and DirectoryDiff in > our protobuf to enable the offline image viewer to parse the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5824) Add a Type field in Snapshot DiffEntry's protobuf definition
Jing Zhao created HDFS-5824: --- Summary: Add a Type field in Snapshot DiffEntry's protobuf definition Key: HDFS-5824 URL: https://issues.apache.org/jira/browse/HDFS-5824 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor We need to add a Type field to differentiate FileDiff and DirectoryDiff in our protobuf to enable the offline image viewer to parse the fsimage. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5781) Use a map to record the mapping between FSEditLogOpCode and the corresponding byte value
[ https://issues.apache.org/jira/browse/HDFS-5781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5781: Attachment: HDFS-5781.002.patch Resubmit the patch to trigger Jenkins. > Use a map to record the mapping between FSEditLogOpCode and the corresponding > byte value > > > Key: HDFS-5781 > URL: https://issues.apache.org/jira/browse/HDFS-5781 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-5781.000.patch, HDFS-5781.001.patch, > HDFS-5781.002.patch, HDFS-5781.002.patch > > > HDFS-5674 uses Enum.values and enum.ordinal to identify an editlog op for a > given byte value. While improving the efficiency, it may cause issue. E.g., > when several new editlog ops are added to trunk around the same time (for > several different new features), it is hard to backport the editlog ops with > larger byte values to branch-2 before those with smaller values, since there > will be gaps in the byte values of the enum. > This jira plans to still use a map to record the mapping between editlog ops > and their byte values. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5799) Make audit logging consistent across ACL APIs.
[ https://issues.apache.org/jira/browse/HDFS-5799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HDFS-5799. - Resolution: Fixed Fix Version/s: HDFS ACLs (HDFS-4685) Hadoop Flags: Reviewed Thanks for the review, Arpit. I committed this to the feature branch. > Make audit logging consistent across ACL APIs. > -- > > Key: HDFS-5799 > URL: https://issues.apache.org/jira/browse/HDFS-5799 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Chris Nauroth > Fix For: HDFS ACLs (HDFS-4685) > > Attachments: HDFS-5799.1.patch > > > Currently, the various ACL APIs are not writing to the audit log > consistently. This patch will ensure that all ACL APIs write to the audit > log and finalize the information that they write. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5241) Provide alternate queuing audit logger to reduce logging contention
[ https://issues.apache.org/jira/browse/HDFS-5241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-5241: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2. > Provide alternate queuing audit logger to reduce logging contention > --- > > Key: HDFS-5241 > URL: https://issues.apache.org/jira/browse/HDFS-5241 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-5241.patch, HDFS-5241.patch > > > The default audit logger has extremely poor performance. The internal > synchronization of log4j causes massive contention between the call handlers > (100 by default) which drastically limits the throughput of the NN. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5823) Document async audit logging
Daryn Sharp created HDFS-5823: - Summary: Document async audit logging Key: HDFS-5823 URL: https://issues.apache.org/jira/browse/HDFS-5823 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp HDFS-5241 added an option for async log4j audit logging. The option is considered semi-experimental and should be documented in hdfs-defaults.xml after it's stability under stress is proven. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5788) listLocatedStatus response can be very large
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-5788: - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for working on the issue, Nathan. I've committed it to trunk and branch-2. > listLocatedStatus response can be very large > > > Key: HDFS-5788 > URL: https://issues.apache.org/jira/browse/HDFS-5788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Fix For: 3.0.0, 2.4.0 > > Attachments: HDFS-5788.patch > > > Currently we limit the size of listStatus requests to a default of 1000 > entries. This works fine except in the case of listLocatedStatus where the > location information can be quite large. As an example, a directory with 7000 > entries, 4 blocks each, 3 way replication - a listLocatedStatus response is > over 1MB. This can chew up very large amounts of memory in the NN if lots of > clients try to do this simultaneously. > Seems like it would be better if we also considered the amount of location > information being returned when deciding how many files to return. > Patch will follow shortly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion
[ https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5754: - Attachment: FeatureInfo.patch > Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion > > > Key: HDFS-5754 > URL: https://issues.apache.org/jira/browse/HDFS-5754 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Brandon Li > Attachments: FeatureInfo.patch, HDFS-5754.001.patch, > HDFS-5754.002.patch, HDFS-5754.003.patch, HDFS-5754.004.patch, > HDFS-5754.006.patch, HDFS-5754.007.patch, HDFS-5754.008.patch > > > Currently, LayoutVersion defines the on-disk data format and supported > features of the entire cluster including NN and DNs. LayoutVersion is > persisted in both NN and DNs. When a NN/DN starts up, it checks its > supported LayoutVersion against the on-disk LayoutVersion. Also, a DN with a > different LayoutVersion than NN cannot register with the NN. > We propose to split LayoutVersion into two independent values that are local > to the nodes: > - NamenodeLayoutVersion - defines the on-disk data format in NN, including > the format of FSImage, editlog and the directory structure. > - DatanodeLayoutVersion - defines the on-disk data format in DN, including > the format of block data file, metadata file, block pool layout, and the > directory structure. > The LayoutVersion check will be removed in DN registration. If > NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling > upgrade, then only rollback is supported and downgrade is not. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5754) Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion
[ https://issues.apache.org/jira/browse/HDFS-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880077#comment-13880077 ] Tsz Wo (Nicholas), SZE commented on HDFS-5754: -- - The patch adds the new system properties (HDFS_SERVICE_LAYOUT etc.) for initializing LayoutVersion.map. It seems that it won't work for unit tests since a test may run both NN and DN but we only have one map. I think we need two maps. Do you agree? - If HdfsConstants.LAYOUT_VERSION is initialized with NameNode.Feature.values() as below, the code in DN using HdfsConstants.LAYOUT_VERSION is incorrect. I guess it needs two LAYOUT_VERSIONs. {code} //HdfsConstants public static final int LAYOUT_VERSION = LayoutVersion - .getCurrentLayoutVersion(); + .getCurrentLayoutVersion(NameNode.Feature.values()); {code} - LayoutFeatureComparator.compare(..) could be simplified as below {code} public int compare(LayoutFeature arg0, LayoutFeature arg1) { return arg0.getLayoutVersion() - arg1.getLayoutVersion(); } {code} - In the patch, the Feature enum in LayoutVersion, NameNode and DataNode are very similar. We could add a new class FeatureInfo for reducing repeated code as shown in FeatureInfo.patch. > Split LayoutVerion into NamenodeLayoutVersion and DatanodeLayoutVersion > > > Key: HDFS-5754 > URL: https://issues.apache.org/jira/browse/HDFS-5754 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Brandon Li > Attachments: HDFS-5754.001.patch, HDFS-5754.002.patch, > HDFS-5754.003.patch, HDFS-5754.004.patch, HDFS-5754.006.patch, > HDFS-5754.007.patch, HDFS-5754.008.patch > > > Currently, LayoutVersion defines the on-disk data format and supported > features of the entire cluster including NN and DNs. LayoutVersion is > persisted in both NN and DNs. When a NN/DN starts up, it checks its > supported LayoutVersion against the on-disk LayoutVersion. Also, a DN with a > different LayoutVersion than NN cannot register with the NN. > We propose to split LayoutVersion into two independent values that are local > to the nodes: > - NamenodeLayoutVersion - defines the on-disk data format in NN, including > the format of FSImage, editlog and the directory structure. > - DatanodeLayoutVersion - defines the on-disk data format in DN, including > the format of block data file, metadata file, block pool layout, and the > directory structure. > The LayoutVersion check will be removed in DN registration. If > NamenodeLayoutVersion or DatanodeLayoutVersion is changed in a rolling > upgrade, then only rollback is supported and downgrade is not. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5343) When cat command is issued on snapshot files getting unexpected result
[ https://issues.apache.org/jira/browse/HDFS-5343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880069#comment-13880069 ] Uma Maheswara Rao G commented on HDFS-5343: --- Thanks for updating the patch Sathish. Here you need to reset the out back with System.out So, you take backup of System.out first and then your code. In finally you just reset with System.out again. {code} ByteArrayOutputStream bao = new ByteArrayOutputStream(); +System.setOut(new PrintStream(bao)); +System.setErr(new PrintStream(bao)); .. {code} Above code should be something like this. {code} PrintStream psBackup = System.out; ByteArrayOutputStream bao = new ByteArrayOutputStream(); +System.setOut(new PrintStream(bao)); +System.setErr(new PrintStream(bao)); try{ .. } finally { System.setOut(psBackup); } {code} Otherwise all SOP from next test onwards may go into the above stream set. Also please keep one empty line gap between testcases. {code} fis.close(); } + /** + * Adding as part of jira HDFS-5343 {code} > When cat command is issued on snapshot files getting unexpected result > -- > > Key: HDFS-5343 > URL: https://issues.apache.org/jira/browse/HDFS-5343 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Reporter: sathish >Assignee: sathish > Attachments: HDFS-5343-0003.patch, HDFS-5343-002.patch > > > first if we create one file with some file length and take the snapshot of > that file,and again append some data through append method to that file,then > if we do cat command operation on snapshot of that file,in general it should > dispaly the data what we added with create operation,but it is displaying the > total data i.e. create +_ appended data. > but if we do the same operation and if we read the contents of snapshot file > through input stream it is just displaying the data created in snapshoted > files. > in this the behaviour of cat command and reading through inputstream is > getting different -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5788) listLocatedStatus response can be very large
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880068#comment-13880068 ] Kihwal Lee commented on HDFS-5788: -- The location counting can be off if blocks are under-replicated or over-replicated, but spending more cycles to make it perfect will be a waste. So I am okay with this approach. +1 > listLocatedStatus response can be very large > > > Key: HDFS-5788 > URL: https://issues.apache.org/jira/browse/HDFS-5788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Attachments: HDFS-5788.patch > > > Currently we limit the size of listStatus requests to a default of 1000 > entries. This works fine except in the case of listLocatedStatus where the > location information can be quite large. As an example, a directory with 7000 > entries, 4 blocks each, 3 way replication - a listLocatedStatus response is > over 1MB. This can chew up very large amounts of memory in the NN if lots of > clients try to do this simultaneously. > Seems like it would be better if we also considered the amount of location > information being returned when deciding how many files to return. > Patch will follow shortly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5788) listLocatedStatus response can be very large
[ https://issues.apache.org/jira/browse/HDFS-5788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13880062#comment-13880062 ] Daryn Sharp commented on HDFS-5788: --- For a bit more context, we had about ~6-7k tasks (erroneously) issuing listLocatedStatus. Each limited response was over 1M. The handler attempts a non-blocking write for the response. If the entire response cannot be written, the call is added to the background responder thread. The kernel accepts well below 1M for a non-blocking write so all the responses were added to the responder thread. The call response byte buffers track the position of the last write, thus the entire response buffer is retained until the full response is sent. Re-allocating a buffer with the unsent response will likely introduce additional memory pressure, so the most logical/simplistic change is limiting the response size of the located status. The end result in our case was the heap bloating by over 8G. Full GC kicked in. The NN was unresponsive for up to 5m at a time. Each time it woke up it marked DNs as dead, causing a flurry of replications which further aggravated the memory issue. Due to other exposed bugs, the NN required a restart. Although more RPCs are required to satisfy the large requests, I believe the tradeoff is reasonable. It's also not likely to be a common occurrence. > listLocatedStatus response can be very large > > > Key: HDFS-5788 > URL: https://issues.apache.org/jira/browse/HDFS-5788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 3.0.0, 0.23.10, 2.2.0 >Reporter: Nathan Roberts >Assignee: Nathan Roberts > Attachments: HDFS-5788.patch > > > Currently we limit the size of listStatus requests to a default of 1000 > entries. This works fine except in the case of listLocatedStatus where the > location information can be quite large. As an example, a directory with 7000 > entries, 4 blocks each, 3 way replication - a listLocatedStatus response is > over 1MB. This can chew up very large amounts of memory in the NN if lots of > clients try to do this simultaneously. > Seems like it would be better if we also considered the amount of location > information being returned when deciding how many files to return. > Patch will follow shortly. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5789) Some of snapshot APIs missing checkOperation double check in fsn
[ https://issues.apache.org/jira/browse/HDFS-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-5789: -- Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Status: Resolved (was: Patch Available) Thanks a lot, Nicholas for the review! I have just committed this to trunk and branch-2. > Some of snapshot APIs missing checkOperation double check in fsn > > > Key: HDFS-5789 > URL: https://issues.apache.org/jira/browse/HDFS-5789 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 3.0.0, 2.3.0 > > Attachments: HDFS-5789.patch > > > HDFS-4591 introduced double checked for HA state while taking fsn lock. > checkoperation made before actually taking lock and after the lock again. > This pattern missed in some of the snapshot APIs and cache management related > APIs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5789) Some of snapshot APIs missing checkOperation double check in fsn
[ https://issues.apache.org/jira/browse/HDFS-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-5789: -- Status: Patch Available (was: Open) > Some of snapshot APIs missing checkOperation double check in fsn > > > Key: HDFS-5789 > URL: https://issues.apache.org/jira/browse/HDFS-5789 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-5789.patch > > > HDFS-4591 introduced double checked for HA state while taking fsn lock. > checkoperation made before actually taking lock and after the lock again. > This pattern missed in some of the snapshot APIs and cache management related > APIs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5789) Some of snapshot APIs missing checkOperation double check in fsn
[ https://issues.apache.org/jira/browse/HDFS-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-5789: - Hadoop Flags: Reviewed +1 patch looks good. > Some of snapshot APIs missing checkOperation double check in fsn > > > Key: HDFS-5789 > URL: https://issues.apache.org/jira/browse/HDFS-5789 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-5789.patch > > > HDFS-4591 introduced double checked for HA state while taking fsn lock. > checkoperation made before actually taking lock and after the lock again. > This pattern missed in some of the snapshot APIs and cache management related > APIs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5318) Support read-only and read-write paths to shared replicas
[ https://issues.apache.org/jira/browse/HDFS-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Sirianni updated HDFS-5318: Attachment: HDFS-5318c-branch-2.patch > Support read-only and read-write paths to shared replicas > - > > Key: HDFS-5318 > URL: https://issues.apache.org/jira/browse/HDFS-5318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Eric Sirianni > Attachments: HDFS-5318.patch, HDFS-5318a-branch-2.patch, > HDFS-5318b-branch-2.patch, HDFS-5318c-branch-2.patch, hdfs-5318.pdf > > > There are several use cases for using shared-storage for datanode block > storage in an HDFS environment (storing cold blocks on a NAS device, Amazon > S3, etc.). > With shared-storage, there is a distinction between: > # a distinct physical copy of a block > # an access-path to that block via a datanode. > A single 'replication count' metric cannot accurately capture both aspects. > However, for most of the current uses of 'replication count' in the Namenode, > the "number of physical copies" aspect seems to be the appropriate semantic. > I propose altering the replication counting algorithm in the Namenode to > accurately infer distinct physical copies in a shared storage environment. > With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor > additional semantics to the {{StorageID}} - namely that multiple datanodes > attaching to the same physical shared storage pool should report the same > {{StorageID}} for that pool. A minor modification would be required in the > DataNode to enable the generation of {{StorageID}} s to be pluggable behind > the {{FsDatasetSpi}} interface. > With those semantics in place, the number of physical copies of a block in a > shared storage environment can be calculated as the number of _distinct_ > {{StorageID}} s associated with that block. > Consider the following combinations for two {{(DataNode ID, Storage ID)}} > pairs {{(DN_A, S_A) (DN_B, S_B)}} for a given block B: > * {{DN_A != DN_B && S_A != S_B}} - *different* access paths to *different* > physical replicas (i.e. the traditional HDFS case with local disks) > ** → Block B has {{ReplicationCount == 2}} > * {{DN_A != DN_B && S_A == S_B}} - *different* access paths to the *same* > physical replica (e.g. HDFS datanodes mounting the same NAS share) > ** → Block B has {{ReplicationCount == 1}} > For example, if block B has the following location tuples: > * {{DN_1, STORAGE_A}} > * {{DN_2, STORAGE_A}} > * {{DN_3, STORAGE_B}} > * {{DN_4, STORAGE_B}}, > the effect of this proposed change would be to calculate the replication > factor in the namenode as *2* instead of *4*. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5789) Some of snapshot APIs missing checkOperation double check in fsn
[ https://issues.apache.org/jira/browse/HDFS-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-5789: -- Attachment: HDFS-5789.patch Attached a simple patch for review. > Some of snapshot APIs missing checkOperation double check in fsn > > > Key: HDFS-5789 > URL: https://issues.apache.org/jira/browse/HDFS-5789 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-5789.patch > > > HDFS-4591 introduced double checked for HA state while taking fsn lock. > checkoperation made before actually taking lock and after the lock again. > This pattern missed in some of the snapshot APIs and cache management related > APIs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5789) Some of snapshot APIs missing checkOperation double check in fsn
[ https://issues.apache.org/jira/browse/HDFS-5789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-5789: -- Summary: Some of snapshot APIs missing checkOperation double check in fsn (was: Some of snapshot, Cache APIs missing checkOperation double check in fsn) > Some of snapshot APIs missing checkOperation double check in fsn > > > Key: HDFS-5789 > URL: https://issues.apache.org/jira/browse/HDFS-5789 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > HDFS-4591 introduced double checked for HA state while taking fsn lock. > checkoperation made before actually taking lock and after the lock again. > This pattern missed in some of the snapshot APIs and cache management related > APIs. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5804) HDFS NFS Gateway fails to mount and proxy when using Kerberos
[ https://issues.apache.org/jira/browse/HDFS-5804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879932#comment-13879932 ] Daryn Sharp commented on HDFS-5804: --- I'm unfamiliar with the nfs code to level set these comments. My initial feeling is that the conditional logic is less than desirable. Relative to the provided patch, I think there's a clean way to avoid the explicit root check. The check seems circumspect as in there shouldn't be a pre-condition that the fuse daemon run as "root". My basic understanding is that fuse runs as root to access user ticket caches. However, there's no reason I couldn't map a different username to uid 0, allow a non-privileged user to access the ticket caches based on group perms, use SELinux capabilities to grant a fsuid of root to the fuse daemon, etc. Anyway, back to the patch. A better way may be to check the given username against the current user. Create a proxy user if they are different, else return the current user. No isSecurityEnabled or root comparison needed. Or better yet, just always create a proxy user. A proxy will work with or w/o security, and proxy of the same user also/should work. I'm unclear how this patch solves the issue of root cannot stat /. A proxy is only being created if the user isn't root so how does this fix the issue? > HDFS NFS Gateway fails to mount and proxy when using Kerberos > - > > Key: HDFS-5804 > URL: https://issues.apache.org/jira/browse/HDFS-5804 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: nfs >Affects Versions: 3.0.0, 2.2.0 >Reporter: Abin Shahab > Attachments: HDFS-5804.patch, javadoc-after-patch.log, > javadoc-before-patch.log > > > When using HDFS nfs gateway with secure hadoop > (hadoop.security.authentication: kerberos), mounting hdfs fails. > Additionally, there is no mechanism to support proxy user(nfs needs to proxy > as the user invoking commands on the hdfs mount). > Steps to reproduce: > 1) start a hadoop cluster with kerberos enabled. > 2) sudo su -l nfsserver and start an nfs server. This 'nfsserver' account has > a an account in kerberos. > 3) Get the keytab for nfsserver, and issue the following mount command: mount > -t nfs -o vers=3,proto=tcp,nolock $server:/ $mount_point > 4) You'll see in the nfsserver logs that Kerberos is complaining about not > having a TGT for root. > This is the stacktrace: > java.io.IOException: Failed on local exception: java.io.IOException: > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[TOKEN, KERBEROS]; Host Details : local host is: > "my-nfs-server-host.com/10.252.4.197"; destination host is: > "my-namenode-host.com":8020; > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764) > at org.apache.hadoop.ipc.Client.call(Client.java:1351) > at org.apache.hadoop.ipc.Client.call(Client.java:1300) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy9.getFileLinkInfo(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileLinkInfo(ClientNamenodeProtocolTranslatorPB.java:664) > at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1713) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileStatus(Nfs3Utils.java:58) > at > org.apache.hadoop.hdfs.nfs.nfs3.Nfs3Utils.getFileAttr(Nfs3Utils.java:79) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.fsinfo(RpcProgramNfs3.java:1643) > at > org.apache.hadoop.hdfs.nfs.nfs3.RpcProgramNfs3.handleInternal(RpcProgramNfs3.java:1891) > at > org.apache.hadoop.oncrpc.RpcProgram.messageReceived(RpcProgram.java:143) > at > org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) > at > org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) > at > org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) > at > org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) > at > org.apache.hadoop.oncrpc.RpcUtil$RpcMessageParserStage.me
[jira] [Updated] (HDFS-5822) InterruptedException to thread sleep ignored
[ https://issues.apache.org/jira/browse/HDFS-5822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ding Yuan updated HDFS-5822: Description: In org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java, there is the following code snippet in the run() method: {noformat} 156: } catch (OutOfMemoryError ie) { 157:IOUtils.cleanup(null, peer); 158:// DataNode can run out of memory if there is too many transfers. 159: // Log the event, Sleep for 30 seconds, other transfers may complete by 160:// then. 161:LOG.warn("DataNode is out of memory. Will retry in 30 seconds.", ie); 162:try { 163: Thread.sleep(30 * 1000); 164:} catch (InterruptedException e) { 165: // ignore 166:} 167: } {noformat} Note that InterruptedException is completely ignored. This might not be safe since any potential events that lead to InterruptedException are lost? More info on why InterruptedException shouldn't be ignored: http://stackoverflow.com/questions/1087475/when-does-javas-thread-sleep-throw-interruptedexception Thanks, Ding was: In org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java, there is the following code snippet in the run() method: 156: } catch (OutOfMemoryError ie) { 157:IOUtils.cleanup(null, peer); 158:// DataNode can run out of memory if there is too many transfers. 159: // Log the event, Sleep for 30 seconds, other transfers may complete by 160:// then. 161:LOG.warn("DataNode is out of memory. Will retry in 30 seconds.", ie); 162:try { 163: Thread.sleep(30 * 1000); 164:} catch (InterruptedException e) { 165: // ignore 166:} 167: } Note that InterruptedException is completely ignored. This might not be safe since any potential events that lead to InterruptedException are lost? More info on why InterruptedException shouldn't be ignored: http://stackoverflow.com/questions/1087475/when-does-javas-thread-sleep-throw-interruptedexception Thanks, Ding > InterruptedException to thread sleep ignored > > > Key: HDFS-5822 > URL: https://issues.apache.org/jira/browse/HDFS-5822 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.2.0 >Reporter: Ding Yuan > > In org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java, there is > the following code snippet in the run() method: > {noformat} > 156: } catch (OutOfMemoryError ie) { > 157:IOUtils.cleanup(null, peer); > 158:// DataNode can run out of memory if there is too many transfers. > 159: // Log the event, Sleep for 30 seconds, other transfers may > complete by > 160:// then. > 161:LOG.warn("DataNode is out of memory. Will retry in 30 seconds.", > ie); > 162:try { > 163: Thread.sleep(30 * 1000); > 164:} catch (InterruptedException e) { > 165: // ignore > 166:} > 167: } > {noformat} > Note that InterruptedException is completely ignored. This might not be safe > since any potential events that lead to InterruptedException are lost? > More info on why InterruptedException shouldn't be ignored: > http://stackoverflow.com/questions/1087475/when-does-javas-thread-sleep-throw-interruptedexception > Thanks, > Ding -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5822) InterruptedException to thread sleep ignored
Ding Yuan created HDFS-5822: --- Summary: InterruptedException to thread sleep ignored Key: HDFS-5822 URL: https://issues.apache.org/jira/browse/HDFS-5822 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.2.0 Reporter: Ding Yuan In org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java, there is the following code snippet in the run() method: 156: } catch (OutOfMemoryError ie) { 157:IOUtils.cleanup(null, peer); 158:// DataNode can run out of memory if there is too many transfers. 159: // Log the event, Sleep for 30 seconds, other transfers may complete by 160:// then. 161:LOG.warn("DataNode is out of memory. Will retry in 30 seconds.", ie); 162:try { 163: Thread.sleep(30 * 1000); 164:} catch (InterruptedException e) { 165: // ignore 166:} 167: } Note that InterruptedException is completely ignored. This might not be safe since any potential events that lead to InterruptedException are lost? More info on why InterruptedException shouldn't be ignored: http://stackoverflow.com/questions/1087475/when-does-javas-thread-sleep-throw-interruptedexception Thanks, Ding -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5821) TestHDFSCLI fails for user names with the dash character
[ https://issues.apache.org/jira/browse/HDFS-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated HDFS-5821: Description: testHDFSConf.xml uses regexes inconsistently to match the username from {code}[a-zA-z0-9]*{code} to {code}[a-z]*{code}. This by far does not cover the space of possible OS user names. For us, it fails for a user name containing {{'-'}}. Instead of keeping updating regex, we propose to use the macro USERNAME. (was: testHDFSConf.xml uses regexes inconsistently to match the username from {{[a-zA-z0-9]*}} to {{[a-z]*}}. This by far does not cover the space of possible OS user names. For us, it fails for a user name containing a {{-}}. Instead of keeping updating regex, we propose to use the macro USERNAME.) > TestHDFSCLI fails for user names with the dash character > > > Key: HDFS-5821 > URL: https://issues.apache.org/jira/browse/HDFS-5821 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.2.0 >Reporter: Gera Shegalov > Attachments: HDFS-5821-trunk.v01.patch > > > testHDFSConf.xml uses regexes inconsistently to match the username from > {code}[a-zA-z0-9]*{code} to {code}[a-z]*{code}. This by far does not cover > the space of possible OS user names. For us, it fails for a user name > containing {{'-'}}. Instead of keeping updating regex, we propose to use the > macro USERNAME. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5821) TestHDFSCLI fails for user names with the dash character
[ https://issues.apache.org/jira/browse/HDFS-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated HDFS-5821: Status: Patch Available (was: Open) > TestHDFSCLI fails for user names with the dash character > > > Key: HDFS-5821 > URL: https://issues.apache.org/jira/browse/HDFS-5821 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.2.0 >Reporter: Gera Shegalov > Attachments: HDFS-5821-trunk.v01.patch > > > testHDFSConf.xml uses regexes inconsistently to match the username from > {{[a-zA-z0-9]*}} to {{[a-z]*}}. This by far does not cover the space of > possible OS user names. For us, it fails for a user name containing a {{-}}. > Instead of keeping updating regex, we propose to use the macro USERNAME. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5821) TestHDFSCLI fails for user names with the dash character
[ https://issues.apache.org/jira/browse/HDFS-5821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated HDFS-5821: Attachment: HDFS-5821-trunk.v01.patch Patch with the proposed fix > TestHDFSCLI fails for user names with the dash character > > > Key: HDFS-5821 > URL: https://issues.apache.org/jira/browse/HDFS-5821 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.2.0 >Reporter: Gera Shegalov > Attachments: HDFS-5821-trunk.v01.patch > > > testHDFSConf.xml uses regexes inconsistently to match the username from > {{[a-zA-z0-9]*}} to {{[a-z]*}}. This by far does not cover the space of > possible OS user names. For us, it fails for a user name containing a {{-}}. > Instead of keeping updating regex, we propose to use the macro USERNAME. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5821) TestHDFSCLI fails for user names with the dash character
Gera Shegalov created HDFS-5821: --- Summary: TestHDFSCLI fails for user names with the dash character Key: HDFS-5821 URL: https://issues.apache.org/jira/browse/HDFS-5821 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.2.0 Reporter: Gera Shegalov testHDFSConf.xml uses regexes inconsistently to match the username from {{[a-zA-z0-9]*}} to {{[a-z]*}}. This by far does not cover the space of possible OS user names. For us, it fails for a user name containing a {{-}}. Instead of keeping updating regex, we propose to use the macro USERNAME. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5776: Attachment: HDFS-5776-v6.txt > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5776: Attachment: (was: HDFS-5776-v6.txt) > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5776: Attachment: HDFS-5776-v6.txt > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776-v6.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core logic is in pread code path, we decide to goto the original > fetchBlockByteRange or the new introduced fetchBlockByteRangeSpeculative per > the above config items. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5776) Support 'hedged' reads in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-5776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13879680#comment-13879680 ] Liang Xie commented on HDFS-5776: - [~arpitagarwal], thanks for your nice review! bq. The concern is too many thread pools created by multiple clients on the same node take it easy, the default configuration: pool=0, that means no extra new threads be created by default. if a end user/application enable hedged read, they should know about this bq. what do you think of not exposing the DFS_DFSCLIENT_HEDGED_READ_THREADPOOL_SIZE setting at all IMHO, i personally prefer the current style, it's less risky, we had a bound queue and once reach the queue limit, we force to exec it in current thread. about the "internal upper bound", how much? 5000? 50? or sth else? i think if enabling this feature explicitly, the end user/application should know a little backgroud at least, right? just like lots of hadoop timeout config parameter, i never see any internal upper bound impl at all... but if you strongly insist on it, i can add. bq. DFSClient#allowHedgedReads seems unnecessary let's keep it there, it's more easier to understand for developer or end user. bq. For DEFAULT_DFSCLIENT_HEDGED_READ_THRESHOLD_MILLIS - can we add an inbuilt minimum delay to defeat applications that set it too low or even zero my opinion is same as the above one. since we don't have any knowledge about end-user's storage configuration, just image if they have a fast flash(with HDFS-2832 enabled), say fusionio, probably one real disk read only cost tens of microseconds, how should we decide a good minimum defeat setting? so i don't like to add it, i totally get your kindly concern:) bq. DFSInputStream#chooseDataNode - can the call to getBestNodeErrorString go inside the "if (failures >=..." clause? another log statement also use it, see "DFSClient.LOG.info("Could not obtain " + block.getBlock...", so it's impossible here. bq. #fetchBlockByteRange - can we rename retVal to something like addressPair? good. let me rename it bq. Do we still need the while loop still there in actualGetFromOneDataNode? yes, but the loop is very very light, only when some exceptions like AccessControlException/InvalidEncryptionKeyException/InvalidBlockTokenException happened, will do extra loop, and all those have a fast quit mechanism, like refetchToken/refetchEncryptionKey or disableLegacyBlockReaderLocal, so this loop will only be executed just a very few times:) bq. There is already a while loop in fetchBlockByteRange enclosing the call to actualGetFromOneDataNode. Now we have a nested loop. In the loop inside fetchBlockByteRange, the responsibily is picking another dn if there's IOException thrown from actualGetFromOneDataNode, so not a fearful nested loop at all, take it easy:) bq. Maybe I misunderstood the code flow but it looks like the way the while loops are nested it defeats the usage of refetchToken and refetchEncryptionKey. It looks like the intention was to limit the refetch to 1 across all retries, now we can refetch multiple times. yes, you had a misunderstanding here. that's why i catch IOException fbae around fetchBlockAt. If we don't catch here, there will be always new refetch from outside loop and will have a spin loop bq. Related to the previous, #actualGetFromOneDataNode, line 1026, - sorry I did not understand why the try-catch was added around the call to fetchBlockAt. hope the above answer could make you clear? hope my poor english doesn't make everything worse, haha:) bq. #actualGetFromOneDataNode, line 1033 - the call to DFSClient.LOG.warn is deleted. Assume that was unintentional? Gd catch! bq. Nitpick - some lines have whitespace-only changes. i found several unnessessiry whitespaces existing, i just removed them to make more clear. Really thanks all for review!!! > Support 'hedged' reads in DFSClient > --- > > Key: HDFS-5776 > URL: https://issues.apache.org/jira/browse/HDFS-5776 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HDFS-5776-v2.txt, HDFS-5776-v3.txt, HDFS-5776-v4.txt, > HDFS-5776-v5.txt, HDFS-5776.txt > > > This is a placeholder of hdfs related stuff backport from > https://issues.apache.org/jira/browse/HBASE-7509 > The quorum read ability should be helpful especially to optimize read outliers > we can utilize "dfs.dfsclient.quorum.read.threshold.millis" & > "dfs.dfsclient.quorum.read.threadpool.size" to enable/disable the hedged read > ability from client side(e.g. HBase), and by using DFSQuorumReadMetrics, we > could export the interested metric valus into client system(e.g. HBase's > regionserver metric). > The core log