James Carter created SPARK-25162:
------------------------------------
Summary: Kubernetes 'in-cluster' client mode and value
spark.driver.host
Key: SPARK-25162
URL: https://issues.apache.org/jira/browse/SPARK-25162
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.4.0
Environment: A java program, deployed to kubernetes, that establishes
a Spark Context in client mode.
Not using spark-submit.
Kubernetes 1.10
AWS EKS
Reporter: James Carter
When creating Kubernetes scheduler 'in-cluster' using client mode, the value
for spark.driver.host can be derived from the IP address of the driver pod.
I observed that the value of _spark.driver.host_ defaulted to the value of
_spark.kubernetes.driver.pod.name_, which is not a valid hostname. This caused
the executors to fail to establish a connection back to the driver.
As a work around, in my configuration I pass the driver's pod name _and_ the
driver's ip address to ensure that executors can establish a connection with
the driver.
_spark.kubernetes.driver.pod.name_ := env.valueFrom.fieldRef.fieldPath:
metadata.name
_spark.driver.host_ := env.valueFrom.fieldRef.fieldPath: status.podIp
Ideally only _spark.kubernetes.driver.pod.name_ need be provided in this
deployment scenario.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
