Siyao Meng created HADOOP-20027:
-----------------------------------
Summary: DatanodeID builds unbracketed IPv6 host:port strings
Key: HADOOP-20027
URL: https://issues.apache.org/jira/browse/HADOOP-20027
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Siyao Meng
Assignee: Siyao Meng
{{DatanodeID}} builds its address strings by plain concatenation:
{code:java}
this.xferAddr = ipAddr + ":" + xferPort; // setIpAndXferPort
return ipAddr + ":" + ipcPort; // getIpcAddr
return ipAddr + ":" + infoPort; // getInfoAddr
return ipAddr + ":" + infoSecurePort; // getInfoSecureAddr
{code}
[DatanodeID.java:138, 204, 211,
218|https://github.com/apache/hadoop/blob/9ede2e95c6fb577d7ae8664833d5e151cbfc47c9/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeID.java#L138]
For a DataNode with an IPv6 address this gives an ambiguous string such as
{{2001:db8:0:0:0:0:0:1:9866}} instead of {{[2001:db8:0:0:0:0:0:1]:9866}}.
{{NetUtils.createSocketAddr}} parses it as the authority of a URI.
{{java.net.URI}} returns a null host for the unbracketed form, so
{{createSocketAddr}} throws {{IllegalArgumentException: Does not contain a
valid host:port authority}}.
{{getXferAddr}} is what the client uses to reach DataNodes: {{DataStreamer}}
for writes, {{DFSInputStream}} for reads, {{DFSUtilClient}} and
{{ClientContext}}. On an IPv6-only cluster that uses IP addresses
({{dfs.client.use.datanode.hostname=false}}, the default), clients cannot
connect to any DataNode. The xfer address is cached in the {{DatanodeID}} and
sent to clients in every block location, so one bad string affects every read
and write. The info addresses have the same problem in the web UI and in
WebHDFS redirects.
{{getXferAddrWithHostname}} and {{getIpcAddrWithHostname}} (lines 225 and 232)
use the same pattern. They are only wrong if the host name is itself an IPv6
literal, but they should use the same helper.
*Proposed fix:* build all of these strings with
{{NetUtils.getHostPortString(host, port)}}, which brackets IPv6 literals since
HADOOP-19958. IPv4 and host name output is unchanged. Add a unit test that
round-trips each address through {{NetUtils.createSocketAddr}} for IPv4, IPv6
and host names.
HADOOP-17543 reports an HDFS put failure on an IPv6 cluster with no stack
trace; this is a likely cause.
POC:
https://github.com/smengcl/hadoop/commit/8aeae47d67401180a954f7859ae423acf641477c
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
