argibbs opened a new pull request, #23647: URL: https://github.com/apache/airflow/pull/23647
# Problem You have an external task which may "skip". When that happens, you want the external task sensor to also skip. This PR enables that behaviour. # Solution The "right" way to do this is: 1. Set `failed_states` to `[State.SKIPPED]` (by default it is `None`) 2. Set `soft_fail=True` With the above settings, when the target task enters the SKIPPED state, the ExternalTaskSensor will see this as a failed state and immediately react to the failure. Because of the `soft_fail` flag, it should not mark as failed, but instead as skipped. # Effect of change Prior to this change, the `ExternalTaskSensor` ignored the `soft_fail` setting, and always marked it as an error. With this change in place, the `ExternalTaskSensor` will instead mark the task as skipped, as desired. # Other comments `soft_fail` would already kick in if the ExternalTaskSensors timed out (i.e. if the external task did not enter an allowed_state or failed_state in the time. That means a possible workaround for the above is to set a timeout on the ExternalTaskSensor which you believe is long enough that the external task should have completed. However (in my experience at least) ExternalTaskSensors are often run without timeouts. This is because if the target task hangs on upstream dependencies, having the external sensor timeout means cascading failures throughout the system which normally means more cleanup. It's usually easier to have the ExternalTaskSensors not timeout and thus only fail if there's an actual problem. Additionally, soft_fail + execution timeouts is not perfect. If the task skips you may want the skip to propagate immediately; under the workaround, the skip would only kick in after the timeout. And of course, there are obvious goldilocks issues with finessing the timeout to be not too short nor too long. # Related Issues closes: #19754 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
