[
https://issues.apache.org/jira/browse/HDFS-7836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339353#comment-14339353
]
Arpit Agarwal commented on HDFS-7836:
-------------------------------------
Thanks for the responses Charles and Colin.
bq. We have seen block reports in the 100MB+ sizes so we suspect that an even
small chunksize than a storage may yield benefits.
IIRC the default 64MB protobuf message limit is hit at 9M blocks. Even with a
hypothetical 10TB disk a low average block size of 10MB, we get 1M blocks/disk
in the foreseeable future. With splitting that gets you to a reasonable ~7MB
block report per disk. I am not saying no to chunking/compression but it would
be useful to see some perf comparison before we add that complexity.
In the past I used CreateEditsLog to generate files and a simple shell script
to generate block files on DataNodes to simulate millions of blocks. Not as
convenient as junit but I'll see if I can clean up and post what I used on
HDFS-7847.
bq. So this proposal is two part. First, use better locking semantics so that
we don't have to take the FSN lock.
bq. Even if we continue to process BRs under one big happy FSN lock, having
multiple threads operate concurrently will yield benefits.
These two sound contradictory. I assume the former is correct and we won't
really take the FSN lock. Also I did not get how you will process one stripe at
a time without repeatedly locking and unlocking, since DataNodes wouldn't know
about the block to stripe mapping to order the reports. I guess I will wait to
see the code.
> BlockManager Scalability Improvements
> -------------------------------------
>
> Key: HDFS-7836
> URL: https://issues.apache.org/jira/browse/HDFS-7836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Charles Lamb
> Assignee: Charles Lamb
> Attachments: BlockManagerScalabilityImprovementsDesign.pdf
>
>
> Improvements to BlockManager scalability.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
