brunocmartins opened a new issue, #58826:
URL: https://github.com/apache/airflow/issues/58826
### Apache Airflow Provider(s)
databricks
### Versions of Apache Airflow Providers
apache-airflow-providers-databricks==7.5.0
### Apache Airflow version
2.10.5
### Operating System
macOS 15.6.1
### Deployment
Astronomer
### Deployment details
Airflow 2.10.5 running on Astronomer Runtime. Standard deployment, no custom
provider patches. Databricks connection configured normally. Reproducible both
in local Docker and in Astronomer-managed environments.
### What happened
When using `DatabricksWorkflowTaskGroup` with `DatabricksNotebookOperator`,
the retry behavior documented for workflow tasks is unavailable because the
UI-level “repair” mechanisms do not function:
- “Repair a single task” button on the notebook task is always disabled and,
when clicked, just reloads the page.
- “Repair All Failed Tasks” button on the workflow `launch` task is disabled
when downstream tasks fail and is only enabled when all downstream tasks
succeed. In such cases, clicking in the button returns: “No tasks to repair.
Not sending repair request.”
- Webserver logs include errors such as:
```
{app.py:1744} ERROR - Exception on /extra_links [GET]
2025-11-28 11:07:17 Traceback (most recent call last):
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
2025-11-28 11:07:17 response = self.full_dispatch_request()
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/flask/app.py", line 1825, in
full_dispatch_request
2025-11-28 11:07:17 rv = self.handle_user_exception(e)
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/flask/app.py", line 1823, in
full_dispatch_request
2025-11-28 11:07:17 rv = self.dispatch_request()
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/flask/app.py", line 1799, in
dispatch_request
2025-11-28 11:07:17 return
self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2025-11-28 11:07:17
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/airflow/www/auth.py", line 250, in
decorated
2025-11-28 11:07:17 return _has_access(
2025-11-28 11:07:17 ^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/airflow/www/auth.py", line 163, in
_has_access
2025-11-28 11:07:17 return func(*args, **kwargs)
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/airflow/utils/session.py", line 97, in
wrapper
2025-11-28 11:07:17 return func(*args, session=session, **kwargs)
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/airflow/www/views.py", line 3250, in
extra_links
2025-11-28 11:07:17 url = task.get_extra_links(ti, link_name)
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/airflow/models/abstractoperator.py",
line 544, in get_extra_links
2025-11-28 11:07:17 return link.get_link(self.unmap(None), ti_key=ti.key)
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 File
"/usr/local/lib/python3.12/site-packages/airflow/providers/databricks/plugins/databricks_workflow.py",
line 493, in get_link
2025-11-28 11:07:17 "tasks_to_repair": task.databricks_task_key,
2025-11-28 11:07:17 ^^^^^^^^^^^^^^^^^^^^^^^^
2025-11-28 11:07:17 AttributeError: 'SerializedBaseOperator' object has no
attribute 'databricks_task_key'
```
According to current provider behavior, tasks inside a Databricks Workflow
are not supposed to use Airflow’s built-in retry mechanism. Instead, retries
must be handled via Databricks’ native task repair, exposed in Airflow through
provider plugins that generate the UI repair buttons.
Because those buttons are non-functional, there is no working way to retry a
Databricks workflow task from Airflow, despite the [documentation indicating
that such repair actions are
supported](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/workflow.html).
### What you think should happen instead
- The “Repair single task” and “Repair all failed tasks” extra-link buttons
should be fully functional for tasks within a `DatabricksWorkflowTaskGroup`.
- Clicking these UI links should:
1. Generate a valid Databricks repair request
2. Repair only the selected notebook task (single-task repair)
3. Repair all failed workflow tasks (multi-task repair)
- The provider should correctly populate and serialize attributes such as
databricks_task_key, run_id, and repair-job paths so the extra links resolve
without error
### How to reproduce
Use the minimal reproducible DAG below. Trigger the DAG, induce a notebook
failure, and attempt to use the UI repair buttons:
```python
from pendulum import datetime
from airflow.models import DAG
from airflow.providers.databricks.operators.databricks_workflow import
DatabricksWorkflowTaskGroup
from airflow.providers.databricks.operators.databricks import
DatabricksNotebookOperator
with DAG(
dag_id="databricks_dag",
start_date=datetime(2025, 11, 1),
schedule=None,
catchup=False,
) as dag:
with DatabricksWorkflowTaskGroup(
group_id="databricks_workflow",
databricks_conn_id="databricks_default",
job_clusters=[
{
"job_cluster_key": "astro-test",
"new_cluster": {
"spark_version": "15.3.x-scala2.12",
"node_type_id": "Standard_DS3_v2",
"num_workers": 1,
},
}
],
notebook_params={"param1": "value1", "param2": "value2"},
notebook_packages=None,
):
DatabricksNotebookOperator(
task_id="run_databricks_notebook",
job_cluster_key="astro-test",
notebook_path="/Workspace/Users/example/airflow_test",
source="WORKSPACE",
wait_for_termination=True,
)
```
Notebook:
```
raise Exception("This is an error message")
```
Steps to reproduce:
1. Initialize an Astronomer project
```
astro dev init --runtime-version 12.10.0
```
2. Create a file under `dags/databricks_dag.py` containing the above DAG code
3. Add the Databricks provider in `requirements.txt`
```
apache-airflow-providers-databricks==7.5.0
```
4. Start Airflow locally (alternatively deploy the project to Astro and
reproduce the issue there)
```
astro dev start
```
5. Configure Databricks, by creating a `databricks_default` connection and
ensuring the notebook defined in `notebook_path` exists in Databricks workspace
6. Trigger the DAG
7. Observe the UI behavior:
- In the failed notebook task, "Repair single tasks" button is disabled or,
when clicked, produces an UI error and does not call Databricks' repair API.
- In the workflow `launch` task, "Repair All Failed Tasks" is disabled (if
any task failed)
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
