amit1510 commented on issue #27830:
URL: https://github.com/apache/airflow/issues/27830#issuecomment-1352633287
Wanted to share some logs. As I got the same error.
Also note the 'Using AAD Token for SPN' msg gets logged 2 times.
[2022-12-14T17:04:16.244+0000] {databricks_base.py:295} INFO - Using AAD
Token for SPN.
[2022-12-14T17:04:16.244+0000] {databricks_base.py:181} INFO - Existing AAD
token is expired, or going to expire soon. Refreshing...
[2022-12-14T17:04:17.170+0000] {databricks.py:75} INFO - Run submitted with
run_id: ********
**[2022-12-14T17:04:17.172+0000] {databricks_base.py:295} INFO - Using AAD
Token for SPN.
[2022-12-14T17:04:17.348+0000] {databricks_base.py:295} INFO - Using AAD
Token for SPN.**
..
..
<
**Then I poll for databricks job status every 30 secs.
And airflow receives the status multiple time as ‘Pending’ (probably cluster
is still starting)
However later I get errors as highlight below**
>
…
..
[2022-12-14T17:11:19.460+0000] {databricks.py:99} INFO - Sleeping for 30
seconds.
[2022-12-14T17:11:49.487+0000] {databricks_base.py:295} INFO - Using AAD
Token for SPN.
[2022-12-14T17:11:49.556+0000] {taskinstance.py:1851} ERROR - Task failed
with exception
Traceback (most recent call last):
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/databricks/hooks/databricks_base.py",
line 347, in _do_api_call
for attempt in self._get_retry_object():
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/tenacity/__init__.py",
line 384, in __iter__
do = self.iter(retry_state=retry_state)
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/tenacity/__init__.py",
line 351, in iter
return fut.result()
File "/opt/bitnami/python/lib/python3.9/concurrent/futures/_base.py", line
439, in result
return self.__get_result()
File "/opt/bitnami/python/lib/python3.9/concurrent/futures/_base.py", line
391, in __get_result
raise self._exception
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/databricks/hooks/databricks_base.py",
line 357, in _do_api_call
response.raise_for_status()
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/requests/models.py",
line 953, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url:
https://adb-************.azuredatabricks.net/api/2.1/jobs/runs/get?run_id=*********
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"/opt/bitnami/airflow/dags/operators/databricks_operator/base_databricks_submit_operator.py",
line 151, in execute
super().execute(context)
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/databricks/operators/databricks.py",
line 339, in execute
_handle_databricks_operator_execution(self, hook, self.log, context)
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/databricks/operators/databricks.py",
line 82, in _handle_databricks_operator_execution
run_state = hook.get_run_state(operator.run_id)
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/databricks/hooks/databricks.py",
line 226, in get_run_state
response = self._do_api_call(GET_RUN_ENDPOINT, json)
File
"/opt/bitnami/airflow/venv/lib/python3.9/site-packages/airflow/providers/databricks/hooks/databricks_base.py",
line 362, in _do_api_call
raise AirflowException(f'Response: {e.response.content}, Status Code:
{e.response.status_code}')
airflow.exceptions.AirflowException: Response:
b'{"error_code":"403","message":"Unauthorized access to Org: **************}’,
Status Code: 403
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
