tsanton opened a new issue, #62433: URL: https://github.com/apache/airflow/issues/62433
### Description AzureContainerInstancesOperator currently blocks a worker slot while it polls Azure Container Instances (ACI) for container group state and fetches logs in a tight loop. For long-running containers, this ties up a regular worker even though the task is mostly waiting on an external service. Similar to the recent work adding deferrable support to AzureBatchOperator, it would be useful to add a deferrable mode to AzureContainerInstancesOperator so it can suspend itself and resume via a triggerer. **Proposed change:** - Add a deferrable: bool = False parameter to AzureContainerInstancesOperator that enables async/deferrable execution. - Add a remove_on_success = True parameter to AzureContainerInstancesOperator. Current behavior deletes the container group in finally; deferrable mode should preserve this. - Logs + XCom: Today the operator streams logs into task logs during execution and optionally pushes logs to XCom. Deferrable mode cannot “live stream” logs in the same way without extra complexity, so we skip live log streaming in deferrable mode and fetch all in execute_complete ### Use case/motivation Adding this parameter and changing the operator to use asynchronous calls would reduce resource consumption when using the Azure Container Instance Operator. ### Related issues Add parameter deferrable to AzureBatchOperator: #59779 Add deferrable support to AzureBatchOperator: #59798 ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
