eladkal commented on code in PR #31840: URL: https://github.com/apache/airflow/pull/31840#discussion_r1228581511
########## docs/apache-airflow/authoring-and-scheduling/deferring.rst: ########## @@ -166,3 +166,13 @@ Airflow tries to only run triggers in one place at once, and maintains a heartbe This means it's possible, but unlikely, for triggers to run in multiple places at once; this is designed into the Trigger contract, however, and entirely expected. Airflow will de-duplicate events fired when a trigger is running in multiple places simultaneously, so this process should be transparent to your Operators. Note that every extra ``triggerer`` you run will result in an extra persistent connection to your database. + + +Difference between Mode='reschedule' and Deferrable=True in Sensors +------------------------------------------------------------------- + +In Airflow, Sensors wait for specific conditions to be met before proceeding with downstream tasks. Sensors have two options for managing idle periods: mode='reschedule' and deferrable=True. + +* Mode='reschedule': The sensor continuously reschedules itself until the condition is met or the maximum retries are reached, suitable for conditions expected to change over time.Resource consumption may be higher as the sensor is repeatedly executed. + +* Deferrable=True: The sensor pauses execution when idle, freeing up worker slots for other tasks, ideal for sensors waiting for external events or resources. Requires proper implementation of deferring logic and coordination with the event or resource. Review Comment: can we structure it similar to what we do with TaskGroup vs SubDAG? https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#taskgroups-vs-subdags to clarify what I mean... when the reader reach this paragraph he is already aware of deferrable and reschedule but might not be very oriented on the specifics . The goal is to mention with 2-3 sentences the power of each on of them then make a compare table by parameters that are easy to comprehend. I don't have specific table structure in mind (it probably require some thinking) but I think it's the best way to compare two capabilities. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
