[
https://issues.apache.org/jira/browse/AIRFLOW-2709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16591614#comment-16591614
]
ASF GitHub Bot commented on AIRFLOW-2709:
-----------------------------------------
betabandido opened a new pull request #3570: [AIRFLOW-2709] Improve error
handling in Databricks hook
URL: https://github.com/apache/incubator-airflow/pull/3570
### JIRA
- [x] My PR addresses the following [Airflow
JIRA](https://issues.apache.org/jira/browse/AIRFLOW-2709) issues and references
them in the PR title.
### Description
- [x] This PR enhances the error handling in the Databricks hook (and in its
corresponding operator). The PR adds the capability to wait between requests,
and it adds support to handle a TEMPORARILY_UNAVAILABLE error. This error is
neither a connection nor timeout error, and thus it is not correctly handled by
the current hook implementation.
### Tests
- [x] My PR adds a unit test to specifically test the new functionality in
the hook, and modifies existing tests in the operator to ensure the hook is
initialized correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Improve error handling in Databricks hook
> -----------------------------------------
>
> Key: AIRFLOW-2709
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2709
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib, hooks
> Reporter: Victor Jimenez
> Priority: Major
>
> The Databricks hook handles both connection and timeout errors. However,
> Databricks sometimes returns a temporarily unavailable error. That error is
> neither a connection nor timeout one. It is just an HTTPError containing the
> following text in the response: TEMPORARILY_UNAVAILABLE. The current error
> handling in the hook should be enhanced to support this error.
> Also, the Databricks hook contains retry logic. Yet, it does not support
> sleeping for some time between requests. This creates a problem in handling
> errors such as the TEMPORARILY_UNAVAILABLE one. This error typically resolves
> after a few seconds. Adding support for sleeping between retry attempts would
> really help to enhance the reliability of Databricks operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
