Siyao Meng created HDFS-18002:
---------------------------------

             Summary: RBF: Router builds unbracketed IPv6 host:port strings for 
heartbeats and the State Store
                 Key: HDFS-18002
                 URL: https://issues.apache.org/jira/browse/HDFS-18002
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: rbf
            Reporter: Siyao Meng
            Assignee: Siyao Meng


The Router builds {{host:port}} strings by concatenation in two places. For an 
IPv6 host the result has no brackets, for example 
{{2001:db8:0:0:0:0:0:10:8020}}, and {{NetUtils.createSocketAddr}} rejects it 
with "Does not contain a valid host:port authority".

*1. NamenodeHeartbeatService with name resolution* 
([NamenodeHeartbeatService.java:206|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java#L206])
{code:java}
rpcAddress = resolvedHost + ":"
    + NetUtils.getPortFromHostPortString(rpcAddress);
serviceAddress = resolvedHost + ":"
    + NetUtils.getPortFromHostPortString(serviceAddress);
{code}
With 
{{dfs.federation.router.monitor.namenode.nameservice.resolution-enabled.<ns>=true}},
 the Router resolves the configured NameNode name to its addresses and creates 
one heartbeat service per address with {{resolvedHost = isa.getHostName()}} 
([Router.java:577|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/Router.java#L577]).
 For an AAAA record with no PTR record that is the IPv6 literal. 
{{lifelineAddress}} and {{webAddress}} are built the same way. {{serviceInit}} 
then creates an {{NNHAServiceTarget}} from {{serviceAddress}}, whose 
{{createSocketAddr}} call throws. The heartbeat service is a child service of 
the Router, so the Router fails to start. {{getRpcAddress()}} has the same 
pattern in its fallback 
([NamenodeHeartbeatService.java:299|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java#L299]),
 reached when a nameservice is monitored without an nnId and lists one 
NameNode. Without resolution the addresses are the configured, bracketed 
strings, and these paths work.

*2. Router admin addresses in the State Store* 
([StateStoreUtils.java:137|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreUtils.java#L137],
 
[StateStoreUtils.java:152|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/store/StateStoreUtils.java#L152])
{code:java}
return hostName + ":" + address.getPort();
...
return inet.getHostAddress() + ":" + address.getPort();
{code}
{{RouterHeartbeatService}} stores each Router's admin address with 
{{getHostPortString}} (default) or {{getIpPortString}} 
({{dfs.federation.router.heartbeat.with.ip.enable=true}}). {{getIpPortString}} 
always gives an unbracketed IPv6 string; {{getHostPortString}} does when there 
is no PTR record. {{getHostPortString}} also only replaces the IPv4 wildcard 
{{0.0.0.0}}, so {{dfs.federation.router.admin-address=[::]:8111}} is stored as 
{{0:0:0:0:0:0:0:0:8111}}. With 
{{dfs.federation.router.mount-table.cache.update=true}} and two or more 
Routers, {{MountTableRefresherService}} passes these addresses to 
{{createSocketAddr}} 
([MountTableRefresherService.java:170|https://github.com/apache/hadoop/blob/1097744bc490b65cad5ab2fbeafff35687971357/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/MountTableRefresherService.java#L170]).
 The {{IllegalArgumentException}} reaches the caller wrapped in Guava's 
{{UncheckedExecutionException}}, which the {{catch (ExecutionException)}} at 
line 248 does not catch, so add, update and remove mount table calls return an 
error after the entry has been written, and the other Routers are not refreshed.

Verified by reading the code, and on JDK 21: {{getHostAddress()}} and the 
no-PTR {{getHostName()}} return unbracketed literals, and a copy of the 
{{createSocketAddr}} parsing rejects them. The Guava wrapping is from the 
documented {{LoadingCache.get}} contract. Not run on a Router or with real DNS.

*Proposed fix:* use {{NetUtils.getHostPortString(host, port)}} for the four 
heartbeat addresses, the {{getRpcAddress()}} fallback, and both 
{{StateStoreUtils}} helpers. In {{getHostPortString}}, treat any 
{{isAnyLocalAddress()}} address as the wildcard. Optionally catch 
{{RuntimeException}} per Router in {{MountTableRefresherService}} so one bad 
record cannot fail the admin call. Add unit tests with IPv6 hosts.

POC: 
https://github.com/smengcl/hadoop/commit/daab9184f1b0818161b3c53a50753aaaefb9f568



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to