[
https://issues.apache.org/jira/browse/HDFS-9014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729222#comment-14729222
]
Hadoop QA commented on HDFS-9014:
---------------------------------
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12754002/ReplicationPolicyFreeSpace.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0ebc658 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12281/console |
This message was automatically generated.
> Block placement policy with respect to DN free space
> ----------------------------------------------------
>
> Key: HDFS-9014
> URL: https://issues.apache.org/jira/browse/HDFS-9014
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: Max Lapan
> Attachments: ReplicationPolicyFreeSpace.patch
>
>
> Default block allocation policy (also known as 'replication policy')
> implemented in NN is random selection from suitable candidates (rack-local or
> 'other rack'). This is ok when all DNs in a cluster has nearly equal amount
> of storage, but leads to problems when some DNs are significantly larger than
> other. In that situation, when NN places new blocks in random fashion, extra
> space becomes almost unusable and, in extreme case can lead to 100% usage of
> all other 'small' DNs with almost empty 'large', which can lead to various
> HDFS and MR problems.
> Situation when we have datanodes of different sizes is quite real in large,
> long-lived systems when different generations of machines are put in a single
> cluster.
> To overcome this, I implemented a different block allocation policy which
> places blocks with respect to free space available on a DN. Please, consider
> it for inclusion in hdfs codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
