steveahnahn opened a new pull request, #69042:
URL: https://github.com/apache/airflow/pull/69042
## Problem
`SageMakerTransformOperator` with `deferrable=True` (which is the default
whenever a deployment sets `[operators] default_deferrable=True`) can mark a
task **successful even though the SageMaker batch-transform job failed**. The
failed job's description is pushed as the XCom result, so downstream tasks run
against missing/invalid transform output with no error surfaced.
## Root cause
The operator defers while the job is still `InProgress`, so the job can fail
*during* the deferred wait. The trigger is `SageMakerTrigger`, which since
#68927 inherits `AwsBaseWaiterTrigger.run()`. On waiter failure that base
**yields** `TriggerEvent({"status": "error", ...})` instead of raising — so
`execute_complete` is resumed with an error event.
`SageMakerTransformOperator.execute_complete` never checked the status: it
logged `"SageMaker job ... completed."` and returned `serialize_result(...)`,
so the task succeeded.
Every other SageMaker operator's `execute_complete` already guards this (`if
status != "success": raise`). Transform was the only one missing it. Before
#68927 the previous trigger *raised* on failure, so the missing guard was
dormant; migrating to the yielding base class turned it into a live
silent-success.
## Fix
Reject a non-success status in
`SageMakerTransformOperator.execute_complete`, matching the sibling operators,
and add a regression test (a deferred transform that fails during the wait now
fails the task instead of reporting success).
**Note on the exception type:** the sibling operators raise
`AirflowException`, but the `check-no-new-airflow-exceptions` hook forbids
*new* `raise AirflowException` usages, so this raises `RuntimeError` (identical
task-failure/retry semantics, and already used elsewhere in this provider, e.g.
`dms.py`).
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (Opus 4.8)
Generated-by: Claude Code (Opus 4.8) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
