o-nikolas opened a new pull request, #73999:
URL: https://github.com/apache/airflow/pull/73999
## Why
EMR Serverless rejects `StopApplication` while any of the application's
sessions is still active, and the provider has no way to terminate a session.
`EmrServerlessStopApplicationOperator` cannot cover the gap — its `force_stop`
argument only calls `cancel_running_jobs`, which cancels job runs and leaves
sessions untouched.
As a result `example_emr_serverless_session`, added in #70763, can never
finish. Its teardown goes straight from `start_session` to `stop_app`, so the
session is still `IDLE` and AWS fails the call:
```
ValidationException: An error occurred (ValidationException) when calling
the StopApplication operation:
Application 00g966g6g91vau0l cannot be STOPPED. All sessions for the
Application need to be in one of
the following states: [TERMINATED, FAILED]
```
`stop_app` fails, `delete_app` fails with it, and the application is left
running and billing.
## What
Adds `EmrServerlessTerminateSessionOperator`, the counterpart to
`EmrServerlessStartSessionOperator`:
- `EmrServerlessHook.terminate_session()` wrapping the `TerminateSession`
API, gated on botocore at runtime by the existing
`_check_interactive_session_support()` so users who do not touch sessions keep
the provider's lower botocore floor.
- A `serverless_session_terminated` waiter. It accepts both `TERMINATED` and
`FAILED` as success, because either state is enough for the application to stop
— that is exactly the condition `StopApplication` enforces.
- `EmrServerlessTerminateSessionTrigger` for deferrable mode.
- The system test now terminates the session before stopping the application.
- Docs note that `force_stop` does not cover sessions, so dag authors know
they need this operator.
`force_stop` is deliberately left alone: it is documented as being about job
runs, and making it terminate sessions as well would change the behaviour of
the existing operator. Happy to extend it in a follow-up if maintainers prefer
that instead.
## Testing
-
`providers/amazon/tests/unit/amazon/aws/{operators/test_emr_serverless_session,hooks/test_emr_serverless,triggers/test_emr}.py`
— 51 passed, covering the sync wait, `wait_for_completion=False`, the
deferrable path, `execute_complete` success/failure, the hook call and its
botocore gate, and trigger serialization.
- Ran the system test against a real AWS account before and after the change.
- Before: `stop_app` failed with the `ValidationException` above,
`delete_app` failed, application left `STARTED`.
- After: all 6 tasks succeeded, `DagRun ... state=success`, and the
application was deleted by the dag's own teardown (`list_applications` shows no
`session-systest` left behind).
<!-- Please keep an empty line before the closing `>` of the following HTML
comment. -->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
