Mridul Muralidharan created SPARK-23720: -------------------------------------------
Summary: Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support Key: SPARK-23720 URL: https://issues.apache.org/jira/browse/SPARK-23720 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.0 Reporter: Mridul Muralidharan In current external shuffle service integration, hostname of the executor and the shuffle service is the same while the port's are different (shuffle service port vs block manager port). When running in non-host networking mode under docker, in yarn, the shuffle service runs on the NM_HOST while the docker container run's under its own (ephemeral and generated) hostname. We should make use of the container's host machine's hostname for shuffle service and not the container hostname, when external shuffle is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org