potiuk commented on code in PR #38100:
URL: https://github.com/apache/airflow/pull/38100#discussion_r1523272699
##########
docs/apache-airflow/authoring-and-scheduling/deferring.rst:
##########
@@ -257,3 +257,24 @@ In Airflow, sensors wait for specific conditions to be met
before proceeding wit
| Built-in functionality for rescheduling | Requires custom
logic to defer task and handle |
| | external changes
|
+--------------------------------------------------------+--------------------------------------------------------+
+
+Difference between ``up_for_retry`` and ``deferred`` state
+-------------------------------------------------------------------
+
+In Airflow, operators that's in `up_for_retry` state will still take worker
slots, becasue the process still runs and does ``sleep`` there. Deferral
Operators
+
+In Airflow when a operator is in the ``up_for_retry`` state, it essentially
means the operator is waiting to be retried after a failure, but it does not
release its resources. The process remains alive, keeping its memory, sockets,
and other resources allocated, except for the CPU. The ``deferred`` state,
utilized only by Deferrable Operators, offers a more sophisticated approach to
handling wait conditions. Deferrable Operators serialize and store the task's
state, freeing all resources. When a condition is met, the task is deserialized
and resumes operation, optimizing resource use by not holding onto resources
during wait periods.
+
++--------------------------------------------------------+--------------------------------------------------------+
+| state='up_for_retry' |
state='deferred' |
++========================================================+========================================================+
+| Keeps resources while waiting. | Releases
resources, pauses execution when idle, |
Review Comment:
Yeah. This would only matter if you use "low-level" retry - such as
`@tenacity` decorators on your external method call - I think that was a bit of
a confusion in our past discussion, where I explained how it works and assumed
the retry you were talking about was those kind of retries, not "task failures"
with retry set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
