Re: ExternalTaskSensor alternatives

2018-08-22 Thread Stefan Seelmann
On 08/22/2018 06:56 PM, Tao Feng wrote: > FYI, there is an existing pr and proposal for improving sensor efficiency( > https://issues.apache.org/jira/browse/AIRFLOW-2747 and > https://github.com/apache/incubator-airflow/pull/3596/files) by the > community. And I hope I'll find some time next week

Re: ExternalTaskSensor alternatives

2018-08-22 Thread Tao Feng
FYI, there is an existing pr and proposal for improving sensor efficiency( https://issues.apache.org/jira/browse/AIRFLOW-2747 and https://github.com/apache/incubator-airflow/pull/3596/files) by the community. For your idea, I am not sure if it is a good idea to add this dag dependent functionality

ExternalTaskSensor alternatives

2018-08-22 Thread Emmanuel Brard
Hi everyone, I've recently looked at the implementation of the ExternalTaskSensor sensor and I was wondering if it would be a good idea to actually implement this check (these checks) at the scheduler level. Basically the ExternalTaskSensor runs a query against the backend database at regular inte