GeoffroyDFox opened a new issue, #33882:
URL: https://github.com/apache/airflow/issues/33882
### Apache Airflow version
2.7.0
### What happened
When using the Deferrable operator DatabricksRunNowOperator, if a task is in
a queue state in Databricks because of a maximum concurrency set directly on
databricks, the operator raises an exception after at least 1 defferal as the
"QUEUED" state is not in the list of supported states on the Databricks Hook
RunState is_terminal property.
Error Snippet :
```
[2023-08-28, 19:39:22 UTC] {task_command.py:410} INFO - Running
<TaskInstance:
trigger_products_categorization_training_databricks.training_tasks.trigger_databricks_training_job
manual__2023-08-28T17:28:04.408402+00:00 map_index=4 [running]> on host
10.4.114.101
[2023-08-28, 19:39:22 UTC] {logging_mixin.py:150} WARNING -
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/mappedoperator.py:615
AirflowProviderDeprecationWarning: `DatabricksRunNowDeferrableOperator` has
been deprecated. Please use
`airflow.providers.databricks.operators.DatabricksRunNowOperator` with
`deferrable=True` instead.
[2023-08-28, 19:39:22 UTC] {pod_generator.py:529} WARNING - Model file
/opt/airflow/pod_templates/pod_template_file.yaml does not exist
[2023-08-28, 19:39:23 UTC] {taskinstance.py:1598} ERROR - Trigger failed:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/triggerer_job_runner.py",
line 537, in cleanup_finished_triggers
result = details["task"].result()
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/triggerer_job_runner.py",
line 615, in run_trigger
async for event in trigger.run():
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/databricks/triggers/databricks.py",
line 84, in run
if run_state.is_terminal:
File
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/databricks/hooks/databricks.py",
line 77, in is_terminal
raise AirflowException(
airflow.exceptions.AirflowException: Unexpected life cycle state: QUEUED: If
the state has been introduced recently, please check the Databricks user guide
for troubleshooting information
```
### What you think should happen instead
As the "QUEUED" state of a job run on Databricks is not a terminal state,
the pod should be deffered using the usual logic.
### How to reproduce
```python
import pendulum, os
from itertools import chain
from airflow import DAG
from airflow.decorators import task
from airflow.providers.databricks.operators.databricks import
DatabricksRunNowOperator
DAG_NAME = "myDag"
default_args = {
"owner": "airflow",
"start_date": pendulum.today("UTC").add(days=-7),
"retries": 0,
"retry_exponential_backoff": False,
}
with DAG(
DAG_NAME,
default_args=default_args,
catchup=False,
schedule=None,
) as dag:
myop = DatabricksRunNowOperator(
task_id='trigger_run_job_op',
dag=dag,
databricks_conn_id='databricks_conn',
job_id=<myJobId>,
python_params=python_params,
polling_period_seconds=300,
deferrable=True,
dag = dag,
)
myop
```
### Operating System
Mac OS Ventura
### Versions of Apache Airflow Providers
apache-airflow-providers-databricks==4.4.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
Code to rework appears to be here :
https://github.com/apache/airflow/blob/ffc9854a81ce3195b2f3e8cdeb4ea90462e112f0/airflow/providers/databricks/hooks/databricks.py#L82
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]