Mridul Muralidharan created SPARK-23720:
-------------------------------------------
Summary: Leverage shuffle service when running in non-host
networking mode in hadoop 3 docker support
Key: SPARK-23720
URL: https://issues.apache.org/jira/browse/SPARK-23720
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Mridul Muralidharan
In current external shuffle service integration, hostname of the executor and
the shuffle service is the same while the port's are different (shuffle service
port vs block manager port).
When running in non-host networking mode under docker, in yarn, the shuffle
service runs on the NM_HOST while the docker container run's under its own
(ephemeral and generated) hostname.
We should make use of the container's host machine's hostname for shuffle
service and not the container hostname, when external shuffle is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
