big-c-note commented on issue #45138: URL: https://github.com/apache/airflow/issues/45138#issuecomment-2567784888
I use DatabricksSubmitRun for what it's worth. Because it's serverless, the start up time is super quick, and you get all the lineage in Databricks because it's running natively on Workflows. That's a huge plus to doing it that way. You can write a little script with Databricks sdk to automate task creation. We literally use one task per workflow so we don't have to deal with databricks workflow dependency stuff. And with serverless it works pretty good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
