argibbs opened a new pull request, #29933: URL: https://github.com/apache/airflow/pull/29933
# Context A while back I made a change to the `ExternalTaskSensor` to properly support soft-fail (see #23647). This was so that I could propagate skipped state - by setting `failed_states=[SKIPPED]` as well as `soft_fail=True`, then my external task sensor would go to skipped state when the upstream task skipped. This works beautifully. Except. Sometimes my upstream task would fail. I didn't want the fail to propagate instantly, but if it didn't get fixed, retried and succeeded within a given interval, then I wanted the sensor to fail. But with `soft_fail=True` the timeout would result in the sensor going to skipped, not failed. I couldn't have both things. So I've made the (small) change in this PR to have an explicit `skipped_states` on the ExternalTaskSensor, along with the existing `failed_states` and `allowed_states`. It does exactly what you think - if the target task enters a state that's in the `skipped_states` list then the sensor will go to skipped. By default the list is empty, so behaviour is unchanged from before. If you are monitoring multiple tasks, then I made behaviour mimic the `failed_states` list. If any one of the tasks goes to a skipped state, then the sensor goes to skipped - unless a different task entered a failed state at the same time, then it will fail. # Changes 1. New `skipped_states` list on the ExternalTaskSensor 2. Tweaks to the `__init__` to handle validation across all three lists (allowed, skipped, failed) 3. Logic to skip if the target reaches a skipped state 4. Test of the new behaviour 5. Newsfragment describing the change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
