jroachgolf84 commented on code in PR #68067:
URL: https://github.com/apache/airflow/pull/68067#discussion_r3392849785
##########
providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py:
##########
@@ -140,6 +145,11 @@ class SparkSubmitOperator(ResumableJobMixin, BaseOperator):
# YARN application ID, K8s driver pod name).
external_id_key = "spark_job_id"
+ # Used only for k8s cluster mode. Caches the pod phase ("Succeeded" /
"Failed") to task_store at the end of
+ # poll_until_complete. On retry, get_job_status reads this before querying
the K8s API
+ # so that a completed job can be identified even after the driver pod is
garbage collected.
+ _K8S_DRIVER_STATUS_KEY = "k8s_driver_status"
Review Comment:
Nit picking, but should this match the case of the parameters (like
`external_id_key`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
