[
https://issues.apache.org/jira/browse/HDFS-8901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15010107#comment-15010107
]
Kai Zheng commented on HDFS-8901:
---------------------------------
bq. There's no bytebuffer version of pread API.
Yeah right. For the byte array version, the passed byte array will be wrapped
into a heap bytebuffer. When Java coder is used, then the heap bytebuffer will
match the bytebuffer type the coder desires.
bq. We can also encode many times stripe by stripe like stateful read, then
data copy isn't needed.
In your illustration, given the buf is of 64k 64k 64k 64k 64k 64k 64k 64k 64k
... (+9 cells), in this change, the whole buffer is divided into +9 cells
wrapping into +9 bytebuffers, these bytebuffers can be referenced into
*decodeInputs* and used for decoding. These derived buffers are not affected in
content. The +9 cells are divided into 3 strips, each strip being decoded a
time. Is there anything missed here? Please clarify. Thanks.
bq. We are not pursuing fewer times of data copy since the performance is
bounded by IO speed and coding speed.
Indicated by our latest benchmark tests, coding computation isn't the concern.
In high speed network and disk, in some case (before hitting the limit) IO may
also not be the bottleneck. I agree data copy isn't the key goal we're working
here, I just did it and don't find the overhead in the way.
> Use ByteBuffer in striping positional read
> ------------------------------------------
>
> Key: HDFS-8901
> URL: https://issues.apache.org/jira/browse/HDFS-8901
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Kai Zheng
> Assignee: Kai Zheng
> Attachments: HDFS-8901-v2.patch, initial-poc.patch
>
>
> Native erasure coder prefers to direct ByteBuffer for performance
> consideration. To prepare for it, this change uses ByteBuffer through the
> codes in implementing striping position read. It will also fix avoiding
> unnecessary data copying between striping read chunk buffers and decode input
> buffers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
