David Mollitor created HDFS-14790:
-------------------------------------
Summary: Support Client Write Fan-Out
Key: HDFS-14790
URL: https://issues.apache.org/jira/browse/HDFS-14790
Project: Hadoop HDFS
Issue Type: Improvement
Components: block placement, hdfs-client
Affects Versions: 3.3.0
Reporter: David Mollitor
The default behavior of an HDFS write is to setup a pipeline. A file is broken
into packets and sent through the pipeline. Pipelining provides good
throughput, but latency suffers.
Allowing a client to specify a fan-out strategy allows the client to send the
packets to the DataNodes concurrently instead of passing the packet through a
pipeline serially.
{code:none}
# Pipeline
C |-------> DN -------> DN -------> DN
# Fan Out
|-------> DN
C |-------> DN
|-------> DN
{code}
Also, if there's a 'min replication' of, for example, 2. The client only needs
to wait for the first 2 ACKs before writing the next packet as long as the 2
ACKs are from different racks. The block placement rules may need to support
this.
HBase requires this improved latency.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
