[
https://issues.apache.org/jira/browse/HDFS-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107026#comment-16107026
]
Weiwei Yang commented on HDFS-12082:
------------------------------------
Hi [~arpitagarwal]
Thanks for looking at this issue. This issue is a bit complex than it looks to
be, please allow me to explain.
Basically this is because following code isn't exactly working as expected,
{code}
this.blockInvalidateLimit = conf.getInt(
DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_KEY, blockInvalidateLimit);
{code}
*expected*:
# Use the value of {{dfs.block.invalidate.limit}} from hdfs-site.xml if set
# If {{dfs.block.invalidate.limit}} is not set in hdfs-site.xml, use max(20 *
HBInterval, 1000)
(like you mentioned)
however in *actual*, it behaves like following
# Use the value of {{dfs.block.invalidate.limit}} from hdfs-site.xml if set
# Use the value of {{dfs.block.invalidate.limit}} in hdfs-default.xml if it is
not set in hdfs-site.xml
it will NEVER return the default value given by argument
{{blockInvalidateLimit}} in cluster env, because we always ship
{{hdfs-default.xml}} in hdfs jar file, that contains property
{{dfs.block.invalidate.limit=1000}}.
The logic in my patch is
Find the bigger value from configuration files (first hdfs-site.xml then back
off to hdfs-default.xml) and compare it with 20*HB_interval, use the bigger one
as the effective value for the invalidate limit. This will ensure that user
won't throttle the block deletion too much for datanodes (even after HB
interval is reconfigured). For example, if HB is 60s, we don't want to let user
to set the limit to less than 1200, otherwise the block deletion will be too
slow.
It might be possible to fix this in another way round, by respecting to its
"original" idea, but that will need to add a method in Configuration class to
tell if a property is configured by user (use getPropertySources?). A bit over
complex?
Please let me know your thought.
Thanks
> BlockInvalidateLimit value is incorrectly set after namenode heartbeat
> interval reconfigured
> ---------------------------------------------------------------------------------------------
>
> Key: HDFS-12082
> URL: https://issues.apache.org/jira/browse/HDFS-12082
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs, namenode
> Reporter: Weiwei Yang
> Assignee: Weiwei Yang
> Attachments: HDFS-12082.001.patch, HDFS-12082.002.patch,
> HDFS-12082.003.patch
>
>
> HDFS-1477 provides an option to reconfigured namenode heartbeat interval
> without restarting the namenode. When the heartbeat interval is reconfigured,
> {{blockInvalidateLimit}} gets recounted
> {code}
> this.blockInvalidateLimit = Math.max(20 * (int) (intervalSeconds),
> DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_DEFAULT);
> {code}
> this doesn't honor the existing value set by {{dfs.block.invalidate.limit}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
