lucafurrer opened a new issue, #43269:
URL: https://github.com/apache/airflow/issues/43269
### Apache Airflow Provider(s)
databricks
### Versions of Apache Airflow Providers
6.11.0
### Apache Airflow version
2.9.3
### Operating System
Kubernetes
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Repackaged of airflow image with Oracle driver
### What happened
When using the DatabricksRunNowOperator with `deferrable=True` the
databricks job gets started but the Task fails at the first poll to check the
job status with
```
[2024-10-22, 16:40:13 CEST] {baseoperator.py:1695} ERROR - Trigger failed:
Traceback (most recent call last):
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 529, in cleanup_finished_triggers
result = details["task"].result()
^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/jobs/triggerer_job_runner.py",
line 601, in run_trigger
async for event in trigger.run():
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/triggers/databricks.py",
line 88, in run
run_state = await self.hook.a_get_run_state(self.run_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks.py",
line 450, in a_get_run_state
response = await self._a_do_api_call(GET_RUN_ENDPOINT, json)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py",
line 625, in _a_do_api_call
async for attempt in self._a_get_retry_object():
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py",
line 166, in __anext__
do = await self.iter(retry_state=self._retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/__init__.py",
line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/_utils.py", line
99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line
398, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in
result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in
__get_result
raise self._exception
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/databricks/hooks/databricks_base.py",
line 641, in _a_do_api_call
self.log.debug("Response Status Code: %s", response.status_code)
^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ClientResponse' object has no attribute 'status_code'
```
This was working with version `6.3.0`
It works fine with `deferred=False`
### What you think should happen instead
The task should continue polling until the databricks job has finished.
### How to reproduce
Use a simple DatabricksRunNowOperator with the mentionned version above.
Databricks runs on Azure.
```python
job_run = DatabricksRunNowOperator(task_id="databricks_job",
job_name="TestJob",
databricks_conn_id="databricks",
deferrable=True)
```
### Anything else
Happens every time
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
