[
https://issues.apache.org/jira/browse/HDFS-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17768155#comment-17768155
]
ASF GitHub Bot commented on HDFS-17181:
---------------------------------------
lfrancke opened a new pull request, #6108:
URL: https://github.com/apache/hadoop/pull/6108
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
https://issues.apache.org/jira/browse/HDFS-17181
When calling WebHDFS to create a file it will happily redirect to nodes that
are in maintenance.
The reason is in the `chooseDatanode` method in `NamenodeWebHdfsMethods`
where it will only call the `BlockPlacementPolicy` (which considers all these
edge cases) in case the `remoteAddr` (i.e. the address making the request to
WebHDFS) is also running a DataNode.
In all other cases it just refers to `NetworkTopology#chooseRandom` which
does not consider any of these circumstances (e.g. load, maintenance).
I don't understand the reason to not just always refer to the placement
policy so this PR fixes that
### How was this patch tested?
We tested it in production and ran the unit tests (had some issues with
those so let's see what the builds here say)
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
> WebHDFS not considering whether a DN is good when called from outside the
> cluster
> ---------------------------------------------------------------------------------
>
> Key: HDFS-17181
> URL: https://issues.apache.org/jira/browse/HDFS-17181
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, webhdfs
> Affects Versions: 3.3.6
> Reporter: Lars Francke
> Priority: Major
> Attachments: Test_fix_for_HDFS-171811.patch
>
>
> When calling WebHDFS to create a file (I'm sure the same problem occurs for
> other actions e.g. OPEN but I haven't checked all of them yet) it will
> happily redirect to nodes that are in maintenance.
> The reason is in the
> [{{chooseDatanode}}|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L307C9-L315]
> method in {{NamenodeWebHdfsMethods}} where it will only call the
> {{BlockPlacementPolicy}} (which considers all these edge cases) in case the
> {{remoteAddr}} (i.e. the address making the request to WebHDFS) is also
> running a DataNode.
>
> In all other cases it just refers to
> [{{NetworkTopology#chooseRandom}}|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L342-L343]
> which does not consider any of these circumstances (e.g. load, maintenance).
> I don't understand the reason to not just always refer to the placement
> policy and we're currently testing a patch to do just that.
> I have attached a draft patch for now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
