Junyeopp opened a new issue, #74661: URL: https://github.com/apache/airflow/issues/74661
### Description Add opt-in deferrable execution to DatabricksSqlSensor for Airflow 3, preserving its existing data-readiness semantics: repeatedly execute a SQL query until the returned result is truthy. This request is for Airflow 3 only; it does not request an Airflow 2 backport. ### Use case/motivation A downstream task may need to wait until data is available in a table or a particular partition. A readiness query can complete successfully while returning no rows, so successful statement execution alone does not indicate that the data is ready. With KubernetesExecutor, reschedule mode releases the worker between checks, but subsequent checks require additional task pods. Deferrable readiness polling would avoid repeatedly starting worker pods while waiting for data. The desired behavior is to keep checking the readiness condition in the triggerer and resume the worker when the condition is met. Expected data-not-ready results should remain normal waiting, rather than becoming SQL errors or failed task attempts. Applicable timeout and failure-policy semantics should remain consistent with non-deferrable execution. The existing provider documentation explicitly demonstrates DatabricksSqlSensor as a way to check for data in a table: https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/sql.html#databrickssqlsensor Would maintainers welcome this capability in DatabricksSqlSensor, or prefer a separate data-readiness sensor? ### Related issues - #48368 requested deferrable support for DatabricksSqlSensor and was closed by #49516. - #49516 introduced DatabricksSQLStatementsSensor. It waits for a submitted statement to complete successfully, rather than repeatedly executing a readiness query until data is available. A statement returning no rows can still succeed, so it does not directly replace the documented data-readiness use case. - #33475 discusses consistency between deferrable sensor behavior and BaseSensorOperator attributes. - #74195 and its sensor proposals address observing newer Delta table versions. That is related, but a newer table version does not necessarily satisfy a particular partition or SQL readiness condition. ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
