adiranjan25 opened a new pull request, #75923: URL: https://github.com/apache/airflow/pull/75923
## Summary Fix Spark Standalone driver-status polling so that the retry counter tracks **consecutive failed polls**, rather than accumulating failures across the entire polling lifecycle. ## Problem `SparkSubmitHook._start_driver_status_tracking()` increments `missed_job_status_reports` whenever the driver-status subprocess exits with a nonzero return code. However, the counter was not reset after a successful poll. As a result, intermittent polling failures could eventually exhaust the retry limit even when successful status checks occurred between failures. This could cause a running Spark application to be reported as failed due to nonconsecutive polling errors. ## Changes - Reset `missed_job_status_reports` to `0` when the polling subprocess completes successfully. - Add a regression test covering alternating successful and failed polls, including more than 10 nonconsecutive failures. - Verify that driver-status tracking continues until the driver reaches `FINISHED`. The fix is intentionally limited to the Spark Standalone driver-status polling retry counter. ## Testing All tests in `test_spark_submit.py` passed locally: - **163 passed** - Ruff lint checks passed. - Ruff formatting checks passed. - `git diff --check` passed. ## Expected behavior Intermittent successful polls reset the failure counter, while consecutive failed polls remain subject to the existing retry threshold. ## AI-assisted development disclosure Generative AI assistance: **ChatGPT (OpenAI)** was used to assist with identifying the bug, drafting the fix and regression test, and troubleshooting the local test environment. The changes were tested locally before submission. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
