pan3793 commented on code in PR #46184:
URL: https://github.com/apache/spark/pull/46184#discussion_r1577131302
##########
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesDriverBuilder.scala:
##########
@@ -73,7 +73,8 @@ private[spark] class KubernetesDriverBuilder {
new HadoopConfDriverFeatureStep(conf),
new KerberosConfDriverFeatureStep(conf),
new PodTemplateConfigMapStep(conf),
- new LocalDirsFeatureStep(conf)) ++ userFeatures
+ new LocalDirsFeatureStep(conf),
+ new DriverIngressFeatureStep(conf)) ++ userFeatures
Review Comment:
that's a good point.
yes, any failure during resource creation will fail the job submission at
the current stage, and I did see the ingress creation failure cases internally.
we mitigate it by providing a switch to allow users to disable creating ingress
for certain jobs by `spark.kubernetes.driver.ingress.enabled`.
currently, we create resources in batches,
https://github.com/apache/spark/blob/48e207f4a2192d474f2c0f141b984ef0c36a78c3/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/KubernetesClientApplication.scala#L161-L181
if we want to make some resources optional, we need to add a new batch to
handle them independently. meanwhile, a new method may need to add in
`KubernetesFeatureConfigStep`, something like
```
trait KubernetesFeatureConfigStep {
...
def getAdditionalOptionalKubernetesResources(): Seq[HasMetadata] =
Seq.empty
}
```
@dongjoon-hyun WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
