potiuk commented on PR #27854: URL: https://github.com/apache/airflow/pull/27854#issuecomment-1325253369
I proposed a solution that fixes both problems (I can split them, but wanted to push them first to see what everyone else think). I believe the way it was done was not sustainable (for one _process_output was hard-coded in common.sql and it was written in the way that it would behave very differently for other operators than Databricks (if another operator would add _process_output, it would not work if the result had not 2 columns in the output). That was rather horrible thing to put in the common.sql. The solution that I have (see the code in comment) requires to: * Bump Databricks major version (backwards-incompatible run() method of Hook) * Bump minor version of common.sql (backwards compatible new feature where description is made available for all DBApiHooks and they can implement their own `_process_output` in consistent way. Tests will likely fail - I have not run them - but I wanted to see if that solution is OK for others who are looking and are interested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
