[ https://issues.apache.org/jira/browse/SPARK-38079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ben updated SPARK-38079: ------------------------ Description: *The problem* When you spark-submit to kubernetes in cluster-mode: # Kubernetes creates the driver # Kubernetes creates a configmap that the driver depends on This is a race condition. If the configmap is not created quickly enough, then the driver will fail to start up properly. See [this stackoverflow post|https://stackoverflow.com/a/58508313] for an alternate description of this problem. *To Reproduce* # Download spark 3.2.0 or 3.2.1 from [https://spark.apache.org/downloads.html] # Create an image with {code:java} bin/docker-image-tool.sh{code} # Spark submit one of the examples to some kubernetes instance # Observe the race condition was: When you spark-submit to kubernetes in cluster-mode: # Kubernetes creates the driver # Kubernetes creates a configmap that the driver depends on This is a race condition. If the configmap is not created quickly enough, then the driver will fail to start up properly. See [this stackoverflow post|https://stackoverflow.com/a/58508313] for an alternate description of this problem. To solve, I suggest that spark should wait for the configmap to exist before proceeding. > Not waiting for configmap before starting driver > ------------------------------------------------ > > Key: SPARK-38079 > URL: https://issues.apache.org/jira/browse/SPARK-38079 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 3.2.0, 3.2.1 > Reporter: Ben > Priority: Major > > *The problem* > When you spark-submit to kubernetes in cluster-mode: > # Kubernetes creates the driver > # Kubernetes creates a configmap that the driver depends on > This is a race condition. If the configmap is not created quickly enough, > then the driver will fail to start up properly. > See [this stackoverflow post|https://stackoverflow.com/a/58508313] for an > alternate description of this problem. > > *To Reproduce* > # Download spark 3.2.0 or 3.2.1 from > [https://spark.apache.org/downloads.html] > # Create an image with > {code:java} > bin/docker-image-tool.sh{code} > # Spark submit one of the examples to some kubernetes instance > # Observe the race condition -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org