Hi, I am running Airflow + Spark + AKS (Azure K8s). Sporadically, when I have a spark job complete, my spark-submit process does not notice that the driver has succeeded and continues to track the job as running. Does anyone know how spark-submit process monitors driver processes on k8s? My expectation is that it monitors them by HTTP, but since we actually deleted the driver pod and the spark-submit process continued to show the process as in progress, I am now questioning this assumption. My end goal is to have spark-submit track driver behavior more accurately.
* Marshall NOTE: This communication and any attachments are for the sole use of the intended recipient(s) and may contain confidential and/or privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by replying to this email, and destroy all copies of the original message.
