eskarimov opened a new pull request #19736: URL: https://github.com/apache/airflow/pull/19736
closes: #18999 The PR intends to add deferrable versions of `DatabricksSubmitRunOperator` and `DatabricksRunNowOperator`, which are using [new Airflow functionality introduced with the version 2.2.0](https://airflow.apache.org/docs/apache-airflow/2.2.0/concepts/deferring.html#deferrable-operators-triggers). There're not so many examples of deferrable operators at the moment, so I'd appreciate if we discuss the following: - Should we update existing operators or create new? If we update existing operators right away, we'll break backward compatibility for Airflow versions prior 2.2.0, so I thought it'd be better to introduce new operators at the moment. - Execution timeout for deferrable operators not handled correctly at the moment, will investigate separately under #19382 - If password authentication is used, the pass is shown in logs of the Trigger process. Will need to check. When using token-based auth it's masked correctly. - You can spot that the regular (non-deferrable) operators use shared function `_handle_databricks_operator_execution` for handling the logic of interacting with Databricks API. However, for the newly created operators I'd prefer to sacrifice DRY a bit and keep the logic described completely in `execute` method, for better readability and flexibility. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
