Stefan Seelmann created AIRFLOW-2747:

             Summary: Explicit re-schedule of sensors
                 Key: AIRFLOW-2747
             Project: Apache Airflow
          Issue Type: Improvement
          Components: core, operators
    Affects Versions: 1.9.0
            Reporter: Stefan Seelmann
            Assignee: Stefan Seelmann
             Fix For: 2.0.0

By default sensors block a worker and just sleep between pokes. This is very 
inefficient, especially when there are many long-running sensors.

There is a hacky workaroud by setting a small timeout value and a high retry 
number. But that has drawbacks:
 * Errors throws by sensors are hidden and the sensor retries too often
 * The sensor is retried in a fixed time interval (with optional exponential 
 * There are many attempts and many log files are generated

 I'd like to propose an explicit reschedule mechanism:
 * A new "reschedule" flag for sensors, if set to True it will raise an 
AirflowRescheduleException that causes a reschedule.
 * AirflowRescheduleException contains the (earliest) re-schedule date.
 * Reschedule requests are recorded in new `task_reschedule` table and 
visualized in the Gantt view.
 * A new TI dependency that checks if a sensor task is ready to be re-scheduled.

 * This change is backward compatible. Existing sensors behave like before. But 
it's possible to set the "reschedule" flag.
 * The poke_interval, timeout, and soft_fail parameters are still respected and 
used to calculate the next schedule time.
 * Custom sensor implementations can even define the next sensible schedule 
date by raising AirflowRescheduleException themselves.
 * Existing TimeSensor and TimeDeltaSensor can also be changed to be 
rescheduled when the time is reached.
 * This mechanism can also be used by non-sensor operators (but then the new 
ReadyToRescheduleDep has to be added to deps or BaseOperator).

Design decisions and caveats:
 * When handling AirflowRescheduleException the `try_number` is decremented. 
That means that subsequent runs use the same try number and write to the same 
log file.
 * Sensor TI dependency check now depends on `task_reschedule` table. However 
only the BaseSensorOperator includes the new ReadyToRescheduleDep for now.

Open questions and TODOs:
 * Should a dedicated state `UP_FOR_RESCHEDULE` be used instead of setting the 
state back to `NONE`? This would require more changes in scheduler code and 
especially in the UI, but the state of a task would be more explicit and more 
transparent to the user.
 * Add example/test for a non-sensor operator
 * Document the new feature

This message was sent by Atlassian JIRA

Reply via email to