SameerMesiah97 opened a new pull request, #75915: URL: https://github.com/apache/airflow/pull/75915
**Description** This change adds deferrable execution support to `AthenaSparkOperator`, allowing calculations to be monitored asynchronously without occupying a worker slot. Set `deferrable=True` to enable this behavior. Both execution modes return the same execution metadata and raise an exception if the calculation fails or is canceled. **Rationale** Athena Spark calculations can run for extended periods. Deferrable execution moves polling to the triggerer, releasing the worker slot while the calculation runs. The `waiter_delay` and `waiter_max_attempts` parameters control polling in both execution modes. **Tests** Added operator tests verifying that: - Deferrable execution passes the calculation ID, polling settings, and AWS configuration to the trigger without synchronous polling. - Successful completion returns the expected execution metadata. - Failed and canceled calculations raise `RuntimeError`. - Missing or unsuccessful trigger events and unexpected calculation states request cancellation and raise `RuntimeError`. - Cancellation failures preserve the original trigger error. Added trigger tests verifying that: - The trigger serializes with the expected classpath and calculation ID. - Polling settings and AWS configuration are preserved during serialization. - The trigger can be reconstructed from its serialized arguments. - The trigger uses `AthenaHook` and the expected Spark waiter, API arguments, status queries, and return value. **Example DAGs** Updated the existing Athena Spark system test DAG to exercise synchronous and deferrable execution sequentially within the same session. **Documentation** Added a deferrable execution example, prerequisites, polling behavior, and cancellation limitations to the Athena Spark guide. Updated operator and trigger docstrings. **Backwards Compatibility** The existing synchronous execution path and metadata return structure are preserved. The `deferrable` parameter defaults to the `operators.default_deferrable` configuration value, falling back to `False`. DAG authors can explicitly set `deferrable=False` to retain synchronous execution regardless of that configuration. ###### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: [GPT 6.1] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
