[
https://issues.apache.org/jira/browse/HADOOP-19408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17922177#comment-17922177
]
Chris Nauroth commented on HADOOP-19408:
----------------------------------------
This was visible intermittently during Dataproc cluster creation or stop/start.
At first, the client is configured to connect to the short form of a GCE
hostname, e.g. cnauroth-test-m. If the master isn't up yet, then the connection
fails and retries. Once the master is up, DNS resolution is successful, and it
resolves to the full hostname: cnauroth-test-m.us-central1-a. However, with
HADOOP-18365, hostname changes are disallowed, based on a strict string match:
{code}
2023-12-28 15:04:19,052 WARN org.apache.hadoop.ipc.Client: Address change
detected. Old: cnauroth-test-m:8031 New:
cnauroth-test-m.us-central1-a.c.cnauroth.internal./100.66.3.224:8031
2023-12-28 15:04:19,053 INFO org.apache.hadoop.io.retry.RetryInvocationHandler:
java.io.IOException: DestHost:destPort
cnauroth-test-m.us-central1-a.c.cnauroth.internal.:8031 , LocalHost:localPort
cnauroth-test-w-2/100.66.3.226:0. Failed on local exception:
java.io.IOException: Couldn't set up IO streams:
java.lang.IllegalArgumentException: Hostname must match: cnauroth-test-m:8031
vs cnauroth-test-m.us-central1-a.c.cnauroth.internal./100.66.3.224:8031, while
invoking ResourceTrackerPBClientImpl.registerNodeManager over null. Retrying
after sleeping for 30000ms. Current retry count: 1.
{code}
I don't have a fix forward in mind. I'm currently mitigating this by reverting
HADOOP-18365 in my distro. That reintroduces the problem HADOOP-18365 was
trying to solve, but that's less impactful for me.
Another workaround is to always configure RPC endpoints to point at the
fully-qualified hostname, so that hostname changes like this never surface into
the Hadoop code.
> RPC clients cannot tolerate hostname changes from DNS resolution.
> -----------------------------------------------------------------
>
> Key: HADOOP-19408
> URL: https://issues.apache.org/jira/browse/HADOOP-19408
> Project: Hadoop Common
> Issue Type: Bug
> Components: ipc
> Affects Versions: 3.4.0, 3.3.5
> Reporter: Chris Nauroth
> Priority: Major
>
> HADOOP-15129 changed RPC connections so that if a client is initialized
> before the server boots, it can re-resolve DNS to discover the updated
> address and complete a successful connection. HADOOP-18365 addressed a
> remaining problem with an old IP address remaining in the {{ConnectionId}}
> after the retry. Unfortunately, this has also reintroduced the problem that
> HADOOP-15129 tried to address, because it disallows hostname changes:
> if (!Objects.equals(this.address.getHostName(), address.getHostName()))
> {
> throw new IllegalArgumentException("Hostname must match: " +
> this.address + " vs "
> + address);
> }
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
