fjmacagno commented on issue #48368: URL: https://github.com/apache/airflow/issues/48368#issuecomment-2766456459
Honestly i think both would be useful, a common use-case of ours is "wait for data to land, then run a query to generate a new table". I think though it would be much harder to make the operator deferrable, since it relies on a persistent http connection to run. If I understand correctly its basically a jdbc connection, and terminating it will terminate the job, unless there is something fancy we could do. I would guess the easiest way would be to write an operator on top of the DatabricksSubmitRunOperator or something like it which wraps the sql in a generated spark job. But, thats pretty different compute-wise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
