[
https://issues.apache.org/jira/browse/HADOOP-1702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghu Angadi updated HADOOP-1702:
---------------------------------
Attachment: HADOOP-1702.patch
The attached patch reduces buffer copies on DataNode. DataNode read one
packet and writes it to mirror and the local disk. Each packet contains
io.file.buffer.size of data. This patch
Below: 'b.i.s' is BufferedInputStream and 'b.o.s' is BufferedOutputStream, and
these are
are much larger than typical size of read or write to them.
- Datanode :
-- before :{noformat}
+ ---> b.o.s ---> mirror
socket
|
client socket ---> b.i.s ---> small DataNode buf --|
|
+ ---> b.o.s ---> local
disk
{noformat}
-- after : {noformat}
+ ---> mirror socket
|
client socket ---> Large Datanode buf (Packet) --|
|
+ ---> local disk
{noformat}
- Client :
-- before: Client used 64k packets irrespective of io.file.buffer.size.
So the extra copy for b.o.s was not present if io.file.buffer.size at or
below 64k. But each packet required 2 writes.
-- after : the size of packet is based on io.file.buffer.size and we use one
write to write to datanode socket.
I don't have numbers regd cpu savings. In absolute numbers, for given amount of
data on one DataNode, CPU saved on DataNode should be larger than CPU saved
when same amount is read with HADOOP-2758.
DFSIO benchmark numbers have been very sensitive to buffering (not to CPU)
while writing, we need to show this patch does not negatively affect this
benchmark.
> Reduce buffer copies when data is written to DFS
> ------------------------------------------------
>
> Key: HADOOP-1702
> URL: https://issues.apache.org/jira/browse/HADOOP-1702
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Raghu Angadi
> Assignee: Raghu Angadi
> Fix For: 0.17.0
>
> Attachments: HADOOP-1702.patch
>
>
> HADOOP-1649 adds extra buffering to improve write performance. The following
> diagram shows buffers as pointed by (numbers). Each eatra buffer adds an
> extra copy since most of our read()/write()s match the io.bytes.per.checksum,
> which is much smaller than buffer size.
> {noformat}
> (1) (2) (3) (5)
> +---||----[ CLIENT ]---||----<>-----||---[ DATANODE ]---||--<>-> to Mirror
>
> | (buffer) (socket) | (4)
> | +--||--+
> ===== |
> ===== =====
> (disk) =====
> {noformat}
> Currently loops that read and write block data, handle one checksum chunk at
> a time. By reading multiple chunks at a time, we can remove buffers (1), (2),
> (3), and (5).
> Similarly some copies can be reduced when clients read data from the DFS.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
