dhanusharer opened a new pull request, #75214: URL: https://github.com/apache/airflow/pull/75214
### Description Resolves #74661. Adds a `deferrable` parameter to `DatabricksSqlSensor` for Airflow 3, allowing users to opt into deferrable data-readiness polling while preserving existing polling and data-readiness semantics. Key highlights: - **`DatabricksSqlSensorTrigger`**: Added a dedicated readiness-polling trigger in `airflow.providers.databricks.triggers.databricks` that uses `asyncio.to_thread` to execute synchronous DBAPI queries via `DatabricksSqlHook` without blocking the triggerer event loop. - **Worker fast-path check**: Initial readiness check runs on the worker first. If data is already ready, the task finishes immediately without deferral overhead. - **Resilience & failure policies**: Preserves `timeout`, `soft_fail`, `never_fail`, and `silent_fail`. Dual timeout enforcement via `self.defer(timeout=...)` and internal `end_time` tracking in the trigger ensures resilience across triggerer restarts or HA failover. - **Compatibility**: Retains existing `poke` / `reschedule` execution when `deferrable=False`. Rejects custom non-serializable callables if `deferrable=True`. - **Docs & Tests**: Added documentation in `sql.rst` and comprehensive unit tests for both `DatabricksSqlSensor` and `DatabricksSqlSensorTrigger`. ### Related Issue Closes: #74661 ### Tests - Added unit tests in `providers/databricks/tests/unit/databricks/sensors/test_databricks_sql.py` - Added unit tests in `providers/databricks/tests/unit/databricks/triggers/test_databricks.py` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
