sunchao opened a new pull request, #57938:
URL: https://github.com/apache/spark/pull/57938
### Why are the changes needed?
`RpcAddress` normalizes IPv6 literals so equivalent addresses compare
equally, but `Utils.normalizeIpIfNeeded` currently recognizes only decimal
digits and colons. Consequently, ordinary hexadecimal IPv6 addresses such as
`2001:db8::dead:beef` bypass normalization and are embedded in Spark RPC URLs
without the required brackets.
For example, enabling `spark.kubernetes.executor.useDriverPodIP` with the
driver pod address `2001:DB8:0:0::BEEF` currently produces an invalid executor
driver URL:
```
spark://CoarseGrainedScheduler@2001:DB8:0:0::BEEF:7098
```
The correct canonical URL is:
```
spark://CoarseGrainedScheduler@[2001:db8::beef]:7098
```
This also affects other RPC addresses that use IPv6 literals. The gap was
introduced when SPARK-42173 added IPv6 canonicalization without accepting
hexadecimal letters in its address matcher.
### What changes were proposed in this pull request?
- Accept uppercase and lowercase hexadecimal digits in both portions of the
existing IPv6 address matcher, preserving the existing Guava canonicalization
behavior.
- Extend `RpcAddressSuite` to cover raw and bracketed IPv6 literals,
uppercase and zero-padded representations, trailing hexadecimal segments,
equivalent-address equality, and Spark URL round trips.
- Extend the existing Kubernetes `spark.kubernetes.executor.useDriverPodIP`
regression test to verify that a raw IPv6 driver pod address produces a
correctly bracketed and canonical executor driver URL.
### Does this PR introduce _any_ user-facing change?
Yes. Valid hexadecimal IPv6 RPC addresses are now normalized and bracketed
correctly, allowing Kubernetes executors to connect directly to IPv6 driver pod
addresses. Existing IPv4, hostname, and numeric-only IPv6 behavior is unchanged.
### How was this patch tested?
```bash
build/sbt -Pkubernetes \
'core/testOnly org.apache.spark.rpc.RpcAddressSuite' \
'kubernetes/testOnly
org.apache.spark.deploy.k8s.features.BasicExecutorFeatureStepSuite'
```
- `RpcAddressSuite`: 11 tests passed.
- `BasicExecutorFeatureStepSuite`: 39 tests passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
