[
https://issues.apache.org/jira/browse/HDFS-12905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16282794#comment-16282794
]
Virajith Jalaparti commented on HDFS-12905:
-------------------------------------------
A 2nd attempt at this following offline discussions with [~elgoiri] - the
concern with patch v1 was the changes involved in the {{HeartBeatManager}}.
This creates an implicit dependency between the {{ProvidedStorageMap}} and the
{{HeartBeatManager}}, and any future changes to the {{HeartBeatManager}} has to
reason about how it affects the {{ProvidedStorageMap}}.
Patch v2 limits the changes to the {{ProvidedStorageMap}}, which is now
responsible to reason about Datanodes whose state is not
{{AdminStates.NORMAL}}. This increases the complexity in the implementation of
the {{ProvidedStorageMap}}. In particular, patch v2 has a potentially higher
compute complexity -- to choose a random node, it first looks for nodes that
are in {{AdminStates.NORMAL}} state. If none are found, it then tries to find
nodes whose state is not {{AdminStates.NORMAL}}. An alternate approach could be
to maintain 2 lists, one for nodes whose state is {{AdminStates.NORMAL}} and
one for nodes whose state is not {{AdminStates.NORMAL}}. While this would be
more efficient at choosing nodes (avoid looking at a node twice), leads to more
complex bookkeeping (e.g., use a background thread to move Datanodes between
these lists).
For now, I think this approach seems reasonable assuming most Datanodes are in
the state {{AdminStates.NORMAL}} and few are not. Once HDFS-12848 is completed,
this could be made pluggable and other efficient implementations are possible.
[~elgoiri], [~chris.douglas] - can you take a look?
> [READ] Handle decommissioning and under-maintenance Datanodes with Provided
> storage.
> ------------------------------------------------------------------------------------
>
> Key: HDFS-12905
> URL: https://issues.apache.org/jira/browse/HDFS-12905
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Virajith Jalaparti
> Assignee: Virajith Jalaparti
> Attachments: HDFS-12905-HDFS-9806.001.patch,
> HDFS-12905-HDFS-9806.002.patch
>
>
> {{ProvidedStorageMap}} doesn't keep track of the state of the datanodes with
> Provided storage. As a result, it can return nodes that are being
> decommissioned or under-maintenance even when live datanodes exist. This JIRA
> is to prefer live datanodes to datanodes in other states.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
