Hello, We are using sensors for lightweight but taking long time tasks. It's like monitoring results of some testing executed on remote machine. We would like to see such a task in a DAG but with so much tasks/dags in running state we will hit max number of processes/dag runs (defined in airflow.cfg) soon. Is it possible to have a sensor, which instead of sleep and keep the process would just exit and be rescheduled later? Are we using sensors in wrong manner?
Thanks mC --------------------------------------------------------------------------------------------------------------------------------- I am an Intel employee. All comments and opinions are my own and do not represent the views of Intel. -------------------------------------------------------------------- Intel Technology Poland sp. z o.o. ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN. Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek przegladanie lub rozpowszechnianie jest zabronione. This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.
