JonasDoubleU opened a new issue, #59189: URL: https://github.com/apache/airflow/issues/59189
### Description Upgrade the Airflow Databricks provider to use the Databricks jobs API 2.2 from 2.1. Specifically the GET_RUN_ENDPOINT, to allow fetching of the status of all tasks within a workflow with more than 100 tasks. Consequently, the fetching probably needs to be modifed to handle pagination. ### Use case/motivation We are using Airflow to trigger Databricks workflows, but some of our workflows contain more than 100 tasks. The support to create workflows with more than 100 tasks with the Databricks jobs API was added in version 2.2. Therefore, when fetching the status of the tasks Airflow fails because it uses version 2.1. Ultimately, we cannot use Airflow to trigger these workflows with the current Airflow Databricks provider. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
