[
https://issues.apache.org/jira/browse/HDFS-9011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733188#comment-14733188
]
Yi Liu commented on HDFS-9011:
------------------------------
Thanks [~jingzhao] for working on this. Besides Nicholas' comments.
*1.* In BlockPoolSlice
{code}
+ private void saveReplicas(List<BlockListAsLongs> persistList) {
+ if (persistList == null || persistList.isEmpty()) {
return;
}
File tmpFile = new File(currentDir, REPLICA_CACHE_FILE + ".tmp");
@@ -787,7 +787,9 @@ private void saveReplicas(BlockListAsLongs
blocksListToPersist) {
FileOutputStream out = null;
try {
out = new FileOutputStream(tmpFile);
- blocksListToPersist.writeTo(out);
+ for (BlockListAsLongs blockLists : persistList) {
+ blockLists.writeTo(out);
+ }
{code}
Now we write {{BlockListAsLongs}} *list* to {{REPLICA_CACHE_FILE}}, so we
should also change the logic of {{readReplicasFromCache}}:
{code}
BlockListAsLongs blocksList = BlockListAsLongs.readFrom(inputStream);
{code}
It currently read the first {{BlockListAsLongs}}.
Also in {{saveReplicas}}, if one BlockListAsLongs has 0 number of blocks, it's
better not to persist it, otherwise there is NullPointerException while reading
replicas from cache file.
*2.* We should also change the description about
{{dfs.blockreport.split.threshold}} in hdfs-default.xml
Nits: some line are longer than 80 characters in the patch.
> Support splitting BlockReport of a storage into multiple RPC
> ------------------------------------------------------------
>
> Key: HDFS-9011
> URL: https://issues.apache.org/jira/browse/HDFS-9011
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jing Zhao
> Assignee: Jing Zhao
> Attachments: HDFS-9011.000.patch, HDFS-9011.001.patch,
> HDFS-9011.002.patch
>
>
> Currently if a DataNode has too many blocks (more than 1m by default), it
> sends multiple RPC to the NameNode for the block report, each RPC contains
> report for a single storage. However, in practice we've seen sometimes even a
> single storage can contains large amount of blocks and the report even
> exceeds the max RPC data length. It may be helpful to support sending
> multiple RPC for the block report of a storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
