pan3793 commented on code in PR #46184:
URL: https://github.com/apache/spark/pull/46184#discussion_r1582805042
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala:
##########
@@ -73,7 +73,8 @@ private[spark] class KubernetesDriverBuilder {
new HadoopConfDriverFeatureStep(conf),
new KerberosConfDriverFeatureStep(conf),
new PodTemplateConfigMapStep(conf),
- new LocalDirsFeatureStep(conf)) ++ userFeatures
+ new LocalDirsFeatureStep(conf),
+ new DriverIngressFeatureStep(conf)) ++ userFeatures
Review Comment:
we need a decision here.
option 1: provide a switch `spark.kubernetes.driver.ingress.enabled` to
allow the user to explicitly enable/disable ingress, and fail the job if
ingress creation failed.
pros:
1. behavior consistent with `spark.ui.enabled`;
2. if ingress creation failed, the user knows what happened rather than
confusing why Spark UI is unavailable;
3. code change is simple;
cons:
1. ingress is actually an optional resource, technically, Spark could run
well without ingress.
option 2: treat ingress as an optional resource, ignore ingress creation
failure
pros:
1. Keep the same stability for Spark jobs submission even with ingress
enabled.
cons:
1. ingress resource is not reliable even the user enable ingress explicitly
2. we need to touch the developer interface to achieve such functionalities
(there is one proposal above)
internally, we adopted option 1, and some users explicitly disabled the
ingress because they didn't want the job submission to fail because of ingress
creation failure.
@dongjoon-hyun @yaooqinn @mridulm @LuciferYang please let me know which one
you are preferred.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
