GeoffroyDFox opened a new issue, #33882:
URL: https://github.com/apache/airflow/issues/33882

   ### Apache Airflow version
   
   2.7.0
   
   ### What happened
   
   When using the Deferrable operator DatabricksRunNowOperator, if a task is in 
a queue state in Databricks because of a maximum concurrency set directly on 
databricks, the operator raises an exception after at least 1 defferal as the 
"QUEUED" state is not in the list of supported states on the Databricks Hook 
RunState is_terminal property.
   
   Error Snippet : 
   ```
   [2023-08-28, 19:39:22 UTC] {task_command.py:410} INFO - Running 
<TaskInstance: 
trigger_products_categorization_training_databricks.training_tasks.trigger_databricks_training_job
 manual__2023-08-28T17:28:04.408402+00:00 map_index=4 [running]> on host 
10.4.114.101
   [2023-08-28, 19:39:22 UTC] {logging_mixin.py:150} WARNING - 
/home/airflow/.local/lib/python3.8/site-packages/airflow/models/mappedoperator.py:615
 AirflowProviderDeprecationWarning: `DatabricksRunNowDeferrableOperator` has 
been deprecated. Please use 
`airflow.providers.databricks.operators.DatabricksRunNowOperator` with 
`deferrable=True` instead.
   [2023-08-28, 19:39:22 UTC] {pod_generator.py:529} WARNING - Model file 
/opt/airflow/pod_templates/pod_template_file.yaml does not exist
   [2023-08-28, 19:39:23 UTC] {taskinstance.py:1598} ERROR - Trigger failed:
   Traceback (most recent call last):
   
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/triggerer_job_runner.py",
 line 537, in cleanup_finished_triggers
       result = details["task"].result()
   
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/triggerer_job_runner.py",
 line 615, in run_trigger
       async for event in trigger.run():
   
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/databricks/triggers/databricks.py",
 line 84, in run
       if run_state.is_terminal:
   
     File 
"/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/databricks/hooks/databricks.py",
 line 77, in is_terminal
       raise AirflowException(
   
   airflow.exceptions.AirflowException: Unexpected life cycle state: QUEUED: If 
the state has been introduced recently, please check the Databricks user guide 
for troubleshooting information
   ```
   
   ### What you think should happen instead
   
   As the "QUEUED" state of a job run on Databricks is not a terminal state, 
the pod should be deffered using the usual logic.
   
   ### How to reproduce
   
   ```python
   import pendulum, os
   from itertools import chain
   from airflow import DAG
   from airflow.decorators import task
   from airflow.providers.databricks.operators.databricks import 
DatabricksRunNowOperator
   
   DAG_NAME = "myDag"
   
   
   default_args = {
       "owner": "airflow",
       "start_date": pendulum.today("UTC").add(days=-7),
       "retries": 0,
       "retry_exponential_backoff": False,
   }
   
   
   with DAG(
           DAG_NAME,
           default_args=default_args,
           catchup=False,
           schedule=None,
   ) as dag:
   
       myop = DatabricksRunNowOperator(
               task_id='trigger_run_job_op',
               dag=dag,
               databricks_conn_id='databricks_conn',
               job_id=<myJobId>,
               python_params=python_params,
               polling_period_seconds=300,
               deferrable=True,
               dag = dag,
           )
       myop
    ```
   
   ### Operating System
   
   Mac OS Ventura
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-databricks==4.4.0
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   Code to rework appears to be here : 
https://github.com/apache/airflow/blob/ffc9854a81ce3195b2f3e8cdeb4ea90462e112f0/airflow/providers/databricks/hooks/databricks.py#L82
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to