Lars Francke created HDFS-17181:
-----------------------------------
Summary: WebHDFS not considering whether a DN is good when called
from outside the cluster
Key: HDFS-17181
URL: https://issues.apache.org/jira/browse/HDFS-17181
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, webhdfs
Affects Versions: 3.3.6
Reporter: Lars Francke
When calling WebHDFS to create a file (I'm sure the same problem occurs for
other actions e.g. OPEN but I haven't checked all of them yet) it will happily
redirect to nodes that are in maintenance.
The reason is in the
[`chooseDatanode`|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L307C9-L315]`
method in `NamenodeWebHdfsMethods` where it will only call the
`BlockPlacementPolicy` (which considers all these edge cases) in case the
`remoteAddr` (i.e. the address making the request to WebHDFS) is also running a
DataNode.
In all other cases it just refers to
[`NetworkTopology#chooseRandom`|https://github.com/apache/hadoop/blob/rel/release-3.3.6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L342-L343]
which does not consider any of these.
I don't understand the reason to not just always refer to the placement policy
and we're currently testing a patch to do just that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
