Siyao Meng created HDFS-17999:
---------------------------------

             Summary: WebHDFS NameNode rejects IPv6 excludedatanodes entries 
with NumberFormatException
                 Key: HDFS-17999
                 URL: https://issues.apache.org/jira/browse/HDFS-17999
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: webhdfs
            Reporter: Siyao Meng
            Assignee: Siyao Meng


{{NamenodeWebHdfsMethods.chooseDatanode}} splits each {{excludedatanodes}} 
entry at the first colon, so an IPv6 entry fails with {{NumberFormatException}} 
and the request returns HTTP 400.

[NamenodeWebHdfsMethods.java:298|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L298]
{code:java}
int idx = host.indexOf(':');
...
excludeNode = bm.getDatanodeManager().getDatanodeByXferAddr(
  host.substring(0, idx), Integer.parseInt(host.substring(idx + 1)));
{code}

The WebHDFS client fills this parameter itself. When a DataNode advertises an 
IPv6 literal as its host name, the NameNode builds the redirect with the 
multi-argument {{URI}} constructor 
([NamenodeWebHdfsMethods.java:465|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java#L465]),
 which brackets it: {{http://[2001:db8::1]:9864/webhdfs/v1/...}}. If the client 
then fails to connect to or read from that DataNode, it adds 
{{[2001:db8::1]:9864}} to {{excludedatanodes}} 
([WebHdfsFileSystem.java:744|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L744]
 uses {{URL.getHost()}}, which keeps the brackets; 
[WebHdfsFileSystem.java:2652|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java#L2652]
 uses {{getAuthority()}}). On the retry the NameNode computes {{idx = 5}} and 
calls {{Integer.parseInt("db8::1]:9864")}}:

{code}
java.lang.NumberFormatException: For input string: "db8::1]:9864"
{code}

{{ExceptionHandler}} maps this to 400 Bad Request. The client does not retry a 
{{RemoteException}}, so the read or write fails instead of going to another 
DataNode. Unbracketed entries ({{2001:db8:0:0:0:0:0:1:9866}}) and bare literals 
fail the same way. The client sends the parameter on retries, which happen when 
HA failover is configured or {{dfs.http.client.retry.policy.enabled}} is true. 
A REST caller that passes an IPv6 {{excludedatanodes}} value hits it directly. 
{{ExcludeDatanodesParam}} does not validate the value.

Verified with a standalone JDK 21 program: {{URI}} and {{URL}} produce the 
bracketed authority above, and the same parsing logic throws the exception 
shown. The 400 mapping and the retry behaviour were traced by reading 
{{ExceptionHandler}}, {{NamenodeWebHdfsMethods.doAsExternalCall}} and 
{{RetryPolicies.FailoverOnNetworkExceptionRetry}}. Not run on an IPv6 
MiniDFSCluster.

Stripping the brackets is not enough for an entry to match. {{Host2NodesMap}} 
is keyed by {{DatanodeID.ipAddr}}, which {{DatanodeManager}} sets from 
{{InetAddress.getHostAddress()}} (expanded form, such as 
{{2001:db8:0:0:0:0:0:1}}), so the literal also has to be normalized.

*Proposed fix:* parse each entry bracket-aware. {{[addr]:port}} is an address 
and port. An entry with more than one colon and no brackets is a bare IPv6 
literal with no port. Anything else is {{host:port}} or {{host}}. Normalize IP 
literals with {{InetAddresses.forString(addr).getHostAddress()}} before the 
lookup. Catch {{IllegalArgumentException}} per entry, log it and skip the 
entry, so one malformed entry cannot fail the request. Add a case to 
{{TestWebHdfsDataLocality}} that passes a bracketed IPv6 entry to 
{{chooseDatanode}}.

Possibly related, found by reading only: the client sends the DataNode's HTTP 
port, while {{getDatanodeByXferAddr}} compares the transfer port. That may make 
client-generated exclusions ineffective on IPv4 as well. Not verified.

POC (the NameNode part needs the normalization above; the {{WebHdfsFileSystem}} 
change is not needed): 
https://github.com/smengcl/hadoop/commit/6f3f5e696bbe8da605d5fc6de96cb6c15bcc66bf



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to