[jira] [Created] (HDFS-13756) Implement the FileSystemLinkResolver in an async way
Duo Zhang created HDFS-13756: Summary: Implement the FileSystemLinkResolver in an async way Key: HDFS-13756 URL: https://issues.apache.org/jira/browse/HDFS-13756 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13755) Add retry support for async rpc
Duo Zhang created HDFS-13755: Summary: Add retry support for async rpc Key: HDFS-13755 URL: https://issues.apache.org/jira/browse/HDFS-13755 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13754) Add authentication support for async rpc
Duo Zhang created HDFS-13754: Summary: Add authentication support for async rpc Key: HDFS-13754 URL: https://issues.apache.org/jira/browse/HDFS-13754 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13753) Implement the mkdir API for AsyncDistributedFileSystem
Duo Zhang created HDFS-13753: Summary: Implement the mkdir API for AsyncDistributedFileSystem Key: HDFS-13753 URL: https://issues.apache.org/jira/browse/HDFS-13753 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang As it only need to connect NN, and easy to write a UT to verify it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13643) Implement basic async rpc client
Duo Zhang created HDFS-13643: Summary: Implement basic async rpc client Key: HDFS-13643 URL: https://issues.apache.org/jira/browse/HDFS-13643 Project: Hadoop HDFS Issue Type: Sub-task Components: ipc Reporter: Duo Zhang Implement the basic async rpc client so we can start working on the DFSClient implementation ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-13640) enable ShortCircuit Read on UC block
[ https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HDFS-13640: -- > enable ShortCircuit Read on UC block > > > Key: HDFS-13640 > URL: https://issues.apache.org/jira/browse/HDFS-13640 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.2 >Reporter: Gang Xie >Priority: Major > > The ShortCircuit read is disabled by HDFS-2757 due to the inconsistency of > the block states. With this limitation, some streaming/messaging application > could not benefit the performance improvement from SCR. In our streaming > system whose storage is HDFS, has around 90% read on the last blocks. So, > it's necessary to enable SCR on the last block, especially the app could > ensure the read after the flush. > After look into the original issue in HDFS-2757, it could only happen when > the read beyond the flush or the local datanode is kicked out from the > pipeline. But if there is the data and the visible length of the block covers > the read length, we could still read the data from the block, right? > > I didn't get a completed solution here. Any suggestion could be helpful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13640) s
[ https://issues.apache.org/jira/browse/HDFS-13640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HDFS-13640. -- Resolution: Invalid Accident? > s > - > > Key: HDFS-13640 > URL: https://issues.apache.org/jira/browse/HDFS-13640 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.2 >Reporter: Gang Xie >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-9372) Typo in DataStorage.recoverTransitionRead
Duo Zhang created HDFS-9372:
---
Summary: Typo in DataStorage.recoverTransitionRead
Key: HDFS-9372
URL: https://issues.apache.org/jira/browse/HDFS-9372
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Duo Zhang
{code:title=DataStorage.java}
if (this.initialized) {
LOG.info("DataNode version: " +
HdfsServerConstants.DATANODE_LAYOUT_VERSION
+ " and NameNode layout version: " + nsInfo.getLayoutVersion());
this.storageDirs = new ArrayList(dataDirs.size());
// mark DN storage is initialized
this.initialized = true;
}
{code}
The first if should be {{!this.initialized}} I think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8683) Implement flow control
Duo Zhang created HDFS-8683: --- Summary: Implement flow control Key: HDFS-8683 URL: https://issues.apache.org/jira/browse/HDFS-8683 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang We have implemented read block over http2 on the POC branch. https://github.com/Apache9/hadoop/tree/HDFS-7966-POC The block reader is implemented using jetty. We wrote a testcase to test performance with MiniCluster and we found that the the flow control of HTTP/2 has a big impact on the performance. Window update frame will be delayed if we create many threads to read, and netty will stop sending data if there is no window space left and thus cause a bad impact on performance. Flow control is a built-in feature of HTTP/2. We ignore it does not mean we can bypass it. So I think we need to support this feature at first place. Or at least, find a way to bypass it(maybe a very large initial window size?) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8671) Add client support for http2 stream channels
Duo Zhang created HDFS-8671:
---
Summary: Add client support for http2 stream channels
Key: HDFS-8671
URL: https://issues.apache.org/jira/browse/HDFS-8671
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Duo Zhang
{{Http2StreamChannel}} is introduced in HDFS-8515 but can only be used at
server side.
Now we implement Http2BlockReader using jetty http2-client in the POC branch,
but the final version of jetty 9.3.0 only accepts java8.
So here we plan to extend the functions of {{Http2StreamChannel}} to support
client side usage and then implement Http2BlockReader based on it. And we still
use jetty http2-client to write testcases to ensure that our http2
implementation is valid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8515) Abstract a DTP/2 HTTP/2 server
Duo Zhang created HDFS-8515: --- Summary: Abstract a DTP/2 HTTP/2 server Key: HDFS-8515 URL: https://issues.apache.org/jira/browse/HDFS-8515 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang Assignee: Duo Zhang Discussed in HDFS-8471. https://issues.apache.org/jira/browse/HDFS-8471?focusedCommentId=14568196page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568196 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8472) Implement a new block reader that reads data over HTTP/2
Duo Zhang created HDFS-8472: --- Summary: Implement a new block reader that reads data over HTTP/2 Key: HDFS-8472 URL: https://issues.apache.org/jira/browse/HDFS-8472 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8471) Implement read block over HTTP/2
Duo Zhang created HDFS-8471: --- Summary: Implement read block over HTTP/2 Key: HDFS-8471 URL: https://issues.apache.org/jira/browse/HDFS-8471 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Duo Zhang Assignee: Duo Zhang -- This message was sent by Atlassian JIRA (v6.3.4#6332)
