dongjoon-hyun commented on PR #52927: URL: https://github.com/apache/spark/pull/52927#issuecomment-3501112803
> The primary purpose of the useDriverPodIP feature is to prevent systemic impacts on Spark from Kubernetes CoreDNS failures. Enabling this feature allows the driver to communicate directly using the Pod's IP address without starting a corresponding DNS service. I get it that you want to use in that way. However, as the feature author, I can say that it's not. 😄 I designed this feature as the extension of `spark.kubernetes.allocation.driver.readinessTimeout` and the primary purpose is to overcome the asynchronous CoreDNS setup slowness by running Spark jobs without waiting K8s DNS setup. > Furthermore, users can directly access the Spark UI via the driver Pod's IP address, resulting in a simpler and more stable deployment solution. This is true, but you seems to forget about the reason why K8s Service exists from the beginning. There are many system integrations to expose Spark Driver Web UI via DNS already. Lastly, this PR has very bad side-effects which disallows users to use both `useDriverPodIP` and `DNS` service together. In other words, this PR simply removes the existing Spark features which is provided by orthogonal `useDriverPodIP` and `excludedFeatureSteps`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
