[
https://issues.apache.org/jira/browse/HDFS-12993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16327487#comment-16327487
]
Ajay Kumar edited comment on HDFS-12993 at 1/16/18 6:09 PM:
------------------------------------------------------------
{quote}{{getBlocks()}} is more expensive when there are many smaller blocks.
E.g. more blocks to go through to gather 10GB worth. Doing path exclusion in
such clusters can have a huge performance impact.
{quote}
[~kihwal], Seems blocks smaller than {{dfs.balancer.getBlocks.min-block-size}}
are excluded before this new path exclusion check happens. However i think we
can safely update check for minBlockSize to {{if (curBlock.getNumBytes() <=
getBlocksMinBlockSize)}}.
[~brahmareddy], Some comments:
# {{Balancer#excludedPaths}} doesn't set excludedPaths to
{{BalancerParameters.Builder }}{code} else if
("-excludedPaths".equalsIgnoreCase(args[i])) \{ checkArgument(++i <
args.length, "exclude paths is missing: args = " + Arrays.toString (args));
excludedPaths = args[i]; } {\code}
# Checkstyle issues in BalancerParameters(L109) ,Dispatcher (L107)
# Could you please add Javadoc for {{Dispatcher#getFilteredPaths,
BlockManager#excludeBlocks}}
# Can we move the check for excludePathPrefixes.isEmpty or null before we call
{
Unknown macro: \{BlockManager#excludeBlocks}
}
was (Author: ajayydv):
{quote}{{getBlocks()}} is more expensive when there are many smaller blocks.
E.g. more blocks to go through to gather 10GB worth. Doing path exclusion in
such clusters can have a huge performance impact.
{quote}
[~kihwal], Seems blocks smaller than \{{dfs.balancer.getBlocks.min-block-size}}
are excluded before this new path exclusion check happens. However i think we
can safely update check for minBlockSize to \{{if (curBlock.getNumBytes() <=
getBlocksMinBlockSize)}}.
[~brahmareddy], Some comments:
# {\{Balancer#excludedPaths}} doesn't set excludedPaths to
\{{BalancerParameters.Builder}}\{code}else if
("-excludedPaths".equalsIgnoreCase(args[i])) {
checkArgument(++i < args.length,
"exclude paths is missing: args = " + Arrays.toString
(args));
excludedPaths = args[i];
}\{code}
# Checkstyle issues in BalancerParameters(L109) ,Dispatcher (L107)
# Could you please add Javadoc for {{Dispatcher#getFilteredPaths,
BlockManager#excludeBlocks}}
# Can we move the check for excludePathPrefixes.isEmpty or null before we call
\{{BlockManager#excludeBlocks}}
> Add an option to Balancer for excluding the paths
> -------------------------------------------------
>
> Key: HDFS-12993
> URL: https://issues.apache.org/jira/browse/HDFS-12993
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: balancer & mover
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
> Priority: Major
> Attachments: HDFS-12993.patch
>
>
> *Usecase:* Users/Customers want to keep the important/frequently used
> blocks(where blocks are not written with favoured nodes and block pinning is
> not enabled) when balancer is triggered.This can be useful datacentric jobs
> which will run frequently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
