dimonchik-suvorov opened a new issue, #31898: URL: https://github.com/apache/airflow/issues/31898
### Apache Airflow version main (development) ### What happened After refactoring `DatabricksSqlHook` lost it's ability to run multiple queries in one Airflow connection (basically same Spark Session). Now it doesn't care what the smart [get_conn](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/hooks/databricks_sql.py#L96) returns and simply creates new connection for each query in the queries list. It is wrong and breaks logic like: ``` USE your_database; select from table_in_your_database; ``` In this case it will create first connection and as consequence Databricks session, runs `USE your_database` statement, then recreate connection/session which doesn't know anything about first query and runs `select from table_in_your_database` statement and obviously it fails because second session doesn't know anything about first query... <img width="1543" alt="image" src="https://github.com/apache/airflow/assets/23456894/284b87a7-a183-4e99-b8f5-b7afe1a10036";> This is a breaking change and cost us some time to understand what went wrong after our upgrade from `2.2.0` to `2.5.3` ### What you think should happen instead smart [get_conn](https://github.com/apache/airflow/blob/main/airflow/providers/databricks/hooks/databricks_sql.py#L96) functions should decide when to create new connection and when not to. Otherwise why do we need so complex logic there? ### How to reproduce Try to send (using `DatabricksSqlHook`) to the Databricks Warehouse a string with two queries: ``` USE your_database; select from table_in_your_database; ``` ### Operating System Debian GNU/Linux 11 (bullseye) ### Versions of Apache Airflow Providers `databricks 3.4.0` ### Deployment Other Docker-based deployment ### Deployment details _No response_ ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
