[
https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reassigned HDFS-916:
--------------------------------
Assignee: (was: Todd Lipcon)
> Rewrite DFSOutputStream to use a single thread with NIO
> -------------------------------------------------------
>
> Key: HDFS-916
> URL: https://issues.apache.org/jira/browse/HDFS-916
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
>
> The DFS write pipeline code has some really hairy multi-threaded
> synchronization. There have been a lot of bugs produced by this (HDFS-101,
> HDFS-793, HDFS-915, tens of others) since it's very hard to understand the
> message passing, lock sharing, and interruption properties. The reason for
> the multiple threads is to be able to simultaneously send and receive. If
> instead of using multiple threads, it used nonblocking IO, I think the whole
> thing would be a lot less error prone.
> I think we could do this in two halves: one half is the DFSOutputStream. The
> other half is BlockReceiver. I opened this JIRA first as I think it's simpler
> (only one TCP connection to deal with, rather than an up and downstream)
> Opinions? Am I crazy? I would like to see some agreement on the idea before I
> spend time writing code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
