hunnguye opened a new issue, #71881:
URL: https://github.com/apache/airflow/issues/71881
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.3.1
### What happened and how to reproduce it?
**Issue Description**
Using the Airflow SparkSubmitOperator Airflow submits an application to the
spark endpoint and awaits the status of the completed job. Whether the job
finished successfully or failed in Spark, the Airflow client receives something
like this:
```
[2026-08-20 11:58:01] DEBUG - polling status of spark driver with id
driver-20260820115651-0003
[2026-08-20 11:58:01] DEBUG - Poll driver status cmd: ['spark-submit',
'--master', 'spark://<host>:7077', '--status', 'driver-20260820115651-0003']
[2026-08-20 11:58:02] DEBUG - spark driver status log: 26/08/20 11:58:02
WARN RestSubmissionClient: Unable to connect to server spark://<host>:7077.
...
AirflowException: Failed to poll for the driver status 10 times: returncode
= 1
```
**Reproducible Steps**
* Spark Standalone Cluster deployment
* Spark-submit configuration:
* deploy-mode = cluster
* spark.standalone.submit.waitAppCompletion = true
* Connection configuration:
* connection type: spark-submit
* host: spark://<host>
* port: 7077
1. Start a Dag which contains a task created by SparkSubmitOperator
2. wait until job finished (configure loglevel to DEBUG for more logs)
exactly related to https://github.com/apache/airflow/issues/20949 also
discussed in https://github.com/apache/airflow/discussions/21799
### What you think should happen instead?
**Expected Behaviour**
The Airflow Client, if set up with
`spark.standalone.submit.waitAppCompletion` should poll and receive the correct
status from spark.
Interestingly, if the Spark job succeeded at the first try (on the spark
server) and the Airflow Client fails, if the Airflow Client retries, it creates
a spark status request, which works, gets the corrects spark job status and
sets the Airflow job to compelted successfully
**Analysis and Background**
(Taken from my own comment in #20949)
A Spark Standalone deployment exposes two relevant endpoints:
* 7077: binary endpoint used for submitting applications
* 6066: REST endpoint, which is also used for driver status requests
After submitting an application, SparkSubmitHook polls the driver status
using _build_track_driver_status_command():
https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L671
The implementation currently branches depending on whether the configured
master URL uses port 6066.
If the port is 6066, Airflow performs a REST request using curl. Otherwise,
it polls using `spark-submit --master <url> --status <driver-id>`
The problem is that when the application is submitted through the normal
Spark Standalone binary endpoint (7077), Airflow also uses that master URL for
status polling. However, `spark-submit --status` internally perform a REST
request for the driver's status, which requires the REST endpoint on port 6066.
see:
[RestSubmissionClient.scala](https://github.com/apache/spark/blob/8acb55b6a8447c563c7a87095a737ed1363999cf/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala#L87)
At the same time, Airflow's Spark submission logic only uses the binary
spark-submit mechanism:
https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L650
Therefore, the configured Spark connection needs to point to the binary
endpoint (7077) for submission, while status polling effectively needs the
corresponding REST endpoint (6066).
To add to that, even if the polling url is somehow configurable in Airlfow
to point to port 6066, the spark submit Client will create a hardcoded http
request (see [RestSubmissionClient.scala -
getBaseUrl](https://github.com/apache/spark/blob/8acb55b6a8447c563c7a87095a737ed1363999cf/core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala#L420)).
Especially in our use case, where we try to put the REST endpoint behind
https, we wouldnt be able to access it via spark submit status polling.
**Related development in Airflow 3.3**
I also noticed that recent development in the Spark provider may already
solve part of this problem at the operator level.
In particular, _StandaloneSparkSubmitBackend contains logic for checking the
status of an existing Spark driver:
https://github.com/apache/airflow/blob/f8b8461e8191f88e72ad8c05b248e0385c21db99/providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py#L204
Starting with Airflow 3.3, this is used during task retries to determine
whether a previously submitted Spark application actually succeeded.
From my understanding, this code already derives the appropriate REST status
URL from the configured Spark master URL, even when the master URL points to
the binary endpoint.
If that understanding is correct, perhaps the same logic could also be
reused by SparkSubmitHook for normal driver status polling instead of
determining the polling mechanism based on whether the configured port is 6066.
Our personal workaround is to override some of the spark hook's methods to
inject the correct REST url. Other users probably take a similiar approach. I
am sure though, there is a more elegant method to implement or refactor the
hook to be able to call the REST Endpoint correctly.
**Contributing**
I'm not very familiar with the Airflow codebase or its contribution process,
but if the maintainers agree with my findings, I would be happy to work on a
patch with some guidance.
### Operating System
_No response_
### Deployment
Docker-Compose
### Apache Airflow Provider(s)
apache-spark
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-spark==6.3.1
### Official Helm Chart version
Not Applicable
### Kubernetes Version
_No response_
### Helm Chart configuration
_No response_
### Docker Image customizations
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
