workingDice commented on code in PR #38100:
URL: https://github.com/apache/airflow/pull/38100#discussion_r1523243544
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -257,3 +257,24 @@ In Airflow, sensors wait for specific conditions to be met
before proceeding wit
| Built-in functionality for rescheduling | Requires custom
logic to defer task and handle |
| | external changes
|
+--------------------------------------------------------+--------------------------------------------------------+
+
+Difference between ``up_for_retry`` and ``deferred`` state
+-------------------------------------------------------------------
+
+In Airflow, operators that's in `up_for_retry` state will still take worker
slots, becasue the process still runs and does ``sleep`` there. Deferral
Operators
+
+In Airflow when a operator is in the ``up_for_retry`` state, it essentially
means the operator is waiting to be retried after a failure, but it does not
release its resources. The process remains alive, keeping its memory, sockets,
and other resources allocated, except for the CPU. The ``deferred`` state,
utilized only by Deferrable Operators, offers a more sophisticated approach to
handling wait conditions. Deferrable Operators serialize and store the task's
state, freeing all resources. When a condition is met, the task is deserialized
and resumes operation, optimizing resource use by not holding onto resources
during wait periods.
+
++--------------------------------------------------------+--------------------------------------------------------+
+| state='up_for_retry' |
state='deferred' |
++========================================================+========================================================+
+| Keeps resources while waiting. | Releases
resources, pauses execution when idle, |
Review Comment:
Thank you very much! I will update the documentation based on what you
explained here.
And just to confirm: it means long `retry_delay` wouldn't cause an issue
right? resources and slots are freed when it's waiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
