[
https://issues.apache.org/jira/browse/HDFS-8481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565708#comment-14565708
]
Kai Zheng commented on HDFS-8481:
---------------------------------
bq. it is beneficial to accumulate multiple of them before sending to decode.
Kai Zheng Could probably suggest a threshold size.
In pure coder's point of view, yes it's good to have larger cell size. It's not
clear yet in this case because the bottleneck might not be in the computation,
instead in network traffic and data copying stuffs? My suggestion would be, if
the accumulation is already available then we could have a default threshold
value like 4MB but allowing it to be configurable in future; otherwise leave
the accumulation optimization for future consideration at all. I would prefer
not to do the accumulation in coder caller layer because it's hard. If it's
good to have then we may do it in coder layer in one place, like having a
{{BufferedRawErasureCoder}} layered on existing raw coders, transparent to
callers.
> Erasure coding: remove workarounds in client side stripped blocks recovering
> ----------------------------------------------------------------------------
>
> Key: HDFS-8481
> URL: https://issues.apache.org/jira/browse/HDFS-8481
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Zhe Zhang
> Assignee: Zhe Zhang
> Attachments: HDFS-8481-HDFS-7285.00.patch,
> HDFS-8481-HDFS-7285.01.patch, HDFS-8481-HDFS-7285.02.patch,
> HDFS-8481-HDFS-7285.03.patch, HDFS-8481-HDFS-7285.04.patch
>
>
> After HADOOP-11847 and related fixes, we should be able to properly calculate
> decoded contents.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
