saadbelgi commented on code in PR #71652:
URL: https://github.com/apache/airflow/pull/71652#discussion_r3890573079
##########
providers/amazon/src/airflow/providers/amazon/aws/sensors/emr.py:
##########
@@ -123,7 +124,11 @@ class
EmrServerlessJobSensor(AwsBaseSensor[EmrServerlessHook]):
:param application_id: application_id to check the state of
:param job_run_id: job_run_id to check the state of
- :param target_states: a set of states to wait for, defaults to 'SUCCESS'
+ :param target_states: a set of states to wait for, defaults to
``SUCCESS``. In deferrable mode
+ the waiter waits for a terminal success state regardless of
``target_states``.
Review Comment:
This was done because I was trying to reuse `EmrServerlessStartJobTrigger`
(used in `EmrServerlessStartJobOperator` which doesn't accept `target_states`
and only waits for completion in deferrable mode).
I have created a new trigger class - `EmrServerlessJobSensorTrigger`- to
replicate this sensor's behaviour in non-deferrable mode. Now, it will fail if
it's a failure state (even if it's in target states), and succeed if any state
in the target state is reached. This behaviour is similar to the one
implemented in `poke` method of this sensor.
One callout is that `EmrServerlessJobSensorTrigger` inherits
`AwsBaseWaiterTrigger` and uses the same waiter as
`EmrServerlessStartJobTrigger`, but overrides the failure and success states
dynamically based on the input target states. This is also done in
`airflow.providers.amazon.aws.triggers.mwaa.MwaaDagRunCompletedTrigger` and
`airflow.providers.amazon.aws.triggers.mwaa.MwaaTaskCompletedTrigger`. This is
done here because I could not find a way to use static waiters defined in json
file because target_states input is a set. If this is to be avoided, I would
have to avoid inheriting from `AwsBaseWaiterTrigger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
