XD-DENG commented on PR #37778: URL: https://github.com/apache/airflow/pull/37778#issuecomment-1981784836
> From the description though I'm not sure why this is need? Why can't the dep be set directly on the Operator object? Hi @ashb , The purpose of this change is to control if a TI is schedulable according to custom logics, in scheduler/executor. Adding a custom TI Dep into the Operator object (via task policy) is possible, but it will not achieve the purpose and may result in unexpected side effect: - the Scheduler/Executor does not check the custom TI deps when it makes decision whether a TI is schedulable or not. - the custom TI Dep may be checked when a task is about to start, but that's in the Worker. This means the job may end up in a `fail`/`retry` status. This also means it's much less efficient in `KubernetesExecutor` (because the deps will be check when the worker pod is already launched) - it may also result in Task Deadlock (I shared more details in [my earlier email int he mailing list dicussion](https://lists.apache.org/thread/wbn5fchopycv52f3obzpv6zc5wtrdp63)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
