SameerMesiah97 opened a new pull request, #60904: URL: https://github.com/apache/airflow/pull/60904
**Description** Added best-effort cleanup to `EC2CreateInstanceOperator` to ensure EC2 instances are terminated when failures occur after successful instance creation. Previously, the operator could create an EC2 instance via `RunInstances` and then fail during post-creation steps (for example, when waiting for the instance with `wait_for_completion=True` and `DescribeInstances` permissions are missing). In these cases, the task failed while leaving the EC2 instance running. The operator now attempts to terminate any created instances if an exception is raised after instance creation. Cleanup is performed opportunistically and does not mask or replace the original exception if termination fails. **Rationale** `EC2CreateInstanceOperator` manages the lifecycle of an external infrastructure resource. If instance creation succeeds but subsequent steps fail, leaving the instance running is both surprising and potentially costly. Failures after instance creation can occur for multiple reasons, including IAM permission errors (for example, missing `DescribeInstances`) as well as loss of access to observability or metadata systems used during task execution. In all of these cases, the operator has failed to complete successfully from Airflow’s perspective, and execution state may no longer be reliable. Attempting best-effort cleanup in these scenarios avoids leaving unmanaged EC2 instances running when the task itself has failed, and aligns with the behavior of other Airflow operators that manage external resources. Cleanup failures are logged and do not mask the original exception, preserving existing failure semantics while improving safety. **Tests** * Added a unit test verifying that EC2 instances are terminated when a failure occurs after instance creation (simulated via a waiter error). * Added a unit test ensuring that failures during cleanup do not mask or override the original exception raised by the operator. **Backwards Compatibility** No changes to the public API or operator parameters. Closes: #60903 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
