SameerMesiah97 opened a new pull request, #63614: URL: https://github.com/apache/airflow/pull/63614
**Description** This change introduces deferrable execution support for `AzureSynapseRunPipelineOperator`. Previously, the operator only supported synchronous execution, where a worker process continuously polled the Azure Synapse API until the pipeline run reached a terminal state. This update allows the operator to defer while waiting for pipeline completion. When `deferrable=True`, the operator submits the pipeline run and defers execution to a new `AzureSynapsePipelineTrigger`. The trigger polls the pipeline run status using the asynchronous Synapse hook and emits a completion event when the pipeline reaches a terminal state or when the configured timeout is exceeded. The operator resumes in `execute_complete()` to log success or raise an exception if the pipeline run failed. The implementation closely mirrors the semantics and behavior of the `AzureDataFactoryRunPipelineOperator` when running in deferrable mode. **Rationale** Azure Synapse pipelines can run for extended periods; often hours depending on the activities involved. In synchronous mode the operator continuously polls the pipeline status from a worker process, which can block worker slots for the duration of the run. Deferrable execution offloads this polling to the triggerer process, allowing the task to suspend while waiting for completion. This implementation builds on the asynchronous Synapse hook introduced in PR #62966, which enables the non-blocking SDK interactions required for triggers. This change also maintains consistency with other long-running operators in the Azure provider. The `AzureDataFactoryRunPipelineOperator` already supports deferrable mode, and deferrable support for `AzureContainerInstancesOperator` is currently being introduced. `AzureSynapseRunPipelineOperator` should provide equivalent deferrable capabilities to maintain consistent behavior across Azure services within the provider. **Tests** Added operator tests verifying that: * The operator does not defer when the pipeline run has already reached a terminal state (success or failure). * The operator defers execution when the pipeline run is in an intermediate state. * `execute_complete()` logs success when the trigger reports a successful completion. * `execute_complete()` raises `AzureSynapsePipelineRunException` when the trigger reports an error. Added trigger tests verifying that: * The trigger serializes correctly with the expected classpath and arguments. * `_build_trigger_event()` returns the correct event for terminal states and `None` for intermediate states. * The trigger emits success and failure events when terminal pipeline states are reached during polling. * Exceptions raised during polling result in an error event. **Documentation** Docstrings for `AzureSynapseRunPipelineOperator` and `AzureSynapsePipelineTrigger` have been updated to document deferrable execution and trigger behavior. **Backwards Compatibility** Deferrable execution is **opt-in** and controlled by the `deferrable` parameter, which defaults to `False`. Existing DAGs continue to use synchronous execution unless `deferrable=True` is explicitly enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
