amoghrajesh commented on PR #67118: URL: https://github.com/apache/airflow/pull/67118#issuecomment-4479872750
> > When an Airflow worker running a SparkSubmitOperator task dies mid-execution, the spark driver keeps running on the spark cluster independently: > > I'm not sure that is always true -- it depends on what mode you submit the job as. (My spark knowledge is rusty though) @ashb This is true specifically for `cluster` deploy mode where the driver runs on the cluster independently of the submitter. In `client` mode the driver runs inside the spark-submit process, so killing the worker also kills the driver. The PR targets cluster mode only. Will tighten the PR description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
