Ankurdeewan commented on issue #59779: URL: https://github.com/apache/airflow/issues/59779#issuecomment-3689180235
Looking at the current implementation, `AzureBatchOperator` submits the job and then spends the remaining of its execution synchronously polling Azure Batch. Both pool readiness and task completion are handled via repeated SDK GET/list calls wrapped in sleep loops (`wait_for_all_node_state()` and `wait_for_job_tasks_to_complete()`), with no callbacks, streams, or long-lived connections involved. That control flow is inherently deferrable and effectively matches the AWS Batch operator’s model prior to deferral. Refactoring this to submit once, defer to a trigger for async polling, and resume on a terminal state would remove unnecessary worker blocking while keeping behavior unchanged, with `deferrable=False` remaining the default for compatibility. I’m happy to take this on and implement it following the AWSBatchOperator/BatchJobTrigger pattern, unless there’s a different direction that would be preferred. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
