potiuk commented on PR #27854: URL: https://github.com/apache/airflow/pull/27854#issuecomment-1325643147
Hey @eladkal @vincbeck @alexott @kazanzhy -> I think I nailed the right approach now (including fixing the tests). I figured out the way how to implement a "common" approach while maintaining backwards compatibilty with the previous DatabricksSQLOperator return value (containing the description). This approach will only make DatabricksSQLHook backwards incompatible - it will return (like all other DBApiHooks) results from the cursor, while description will be available as `.last_description' field in the Hook. However DatabricksSQLOperator will retain the original behaviour with returning Tuple[description, results]. This is made possible by changing the semantics of `_process_output` method. The `_process_output` should return the results that are supposed to be returned by "execute" method of the SQLOperator. This way the XCom returned by DatabricksSQLOperator will be the same as previously `(description, results)` rather than just `results`. This way, the DAGs using DatabricksSQLOperator will not be affected. But using DatabricksSQLHook to return the same output as all other DBApiHooks. I think this strikes a very nice balance between "common" and "compatible". WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
