[
https://issues.apache.org/jira/browse/HDFS-8719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619376#comment-14619376
]
Jing Zhao commented on HDFS-8719:
---------------------------------
Looks like switching streamer and refreshing {{chunksPerPacket}}/{{packetSize}}
should always happen together. Do we also need to update the current
{{writeChunk}} function? Also shall we put these two ops into the same function
and always call the combined function?
{code}
if (cellFull) {
int next = index + 1;
//When all data cells in a stripe are ready, we need to encode
//them and generate some parity cells. These cells will be
//converted to packets and put to their DataStreamer's queue.
if (next == numDataBlocks) {
cellBuffers.flipDataBuffers();
writeParityCells();
next = 0;
}
setCurrentStreamer(next);
}
{code}
> Erasure Coding: client generates too many small packets when writing parity
> data
> --------------------------------------------------------------------------------
>
> Key: HDFS-8719
> URL: https://issues.apache.org/jira/browse/HDFS-8719
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Li Bo
> Assignee: Li Bo
> Attachments: HDFS-8719-001.patch, HDFS-8719-HDFS-7285-001.patch,
> HDFS-8719-HDFS-7285-002.patch
>
>
> Typically a packet is about 64K, but when writing parity data, many small
> packets with size 512 bytes are generated. This may slow the write speed and
> increase the network IO.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
