jangByeongHui opened a new pull request, #67794:
URL: https://github.com/apache/airflow/pull/67794
## Summary
Airflow 3.x now ships three mechanisms for handling long-running or blocking
work inside a task — deferrable operators, async Python tasks, and
`ResumableJobMixin` — but no documentation explains the trade-offs or when to
reach for each one.
This PR closes that gap by extending the existing *Deferred vs Async
Operators*
page in the Task SDK docs with a full three-way comparison, and adding a
short
cross-reference in the core deferring guide.
closes: #67706
---
## Changes
### `task-sdk/docs/deferred-vs-async-operators.rst` — primary
- **Page title** updated from *Deferred vs Async Operators* to
*Deferred, Async, and Resumable Operators*.
The RST label `sdk-deferred-vs-async-operators` is preserved unchanged so
existing external links continue to work.
- **Intro paragraph** updated to mention all three patterns.
A `.. versionchanged:: 3.3.0` note is added, pointing readers to the new
section.
- **New "Resumable Operators" section** added before "When not to use…":
- Explains the core mechanic: the mixin persists the external job
identifier
to `task_state` before polling; on retry it reconnects to the
already-running
job instead of submitting a duplicate.
- Lists key characteristics: worker slot is *held* (not freed), no
Triggerer
required, duplicate submission prevented automatically.
- Provides when-to-use and when-to-avoid guidance.
- Includes a full code example showing how to subclass `ResumableJobMixin`,
with a note that `SparkSubmitOperator` uses this pattern in practice.
- **New "Three-way Comparison" table** (`.. list-table::`) comparing
deferrable /
async / resumable across six dimensions:
| Dimension | Deferrable | Async `@task` | Resumable |
|---|---|---|---|
| Worker slot during wait | Freed | Held | Held |
| Requires Triggerer | Yes | No | No |
| State passed on retry | Via `method_name`/`kwargs` | Not persisted |
Auto via `task_state` |
| Duplicate prevention | Manual | Manual | Automatic |
| Ideal workload | Single external event | Many concurrent I/O ops |
Long-running remote job |
| Available from | Airflow 2.2 | Airflow 3.2 | Airflow 3.3 |
- **New bullet** appended to the existing "When **not** to use" section
advising
against resumable operators when a Triggerer is available and the operator
is
being written from scratch.
### `airflow-core/docs/authoring-and-scheduling/deferring.rst` — secondary
- Updated the cross-reference note near the top (line 35) to mention
resumable
operators alongside deferred and async.
- Appended a new `.. _deferring/resumable:` subsection after the existing
`mode='reschedule' vs deferrable=True` comparison table, summarising the
resumable pattern and linking to the Task SDK page for the full comparison.
### `task-sdk/docs/index.rst` — tertiary
- Updated the "Choosing Between" section heading and paragraph to describe
all
three patterns instead of two.
---
## Testing
This PR changes documentation only. No Python source files were modified.
Pre-commit hooks (`prek run --from-ref main --stage pre-commit`) were run
locally and passed — including RST lint, codespell, and newsfragment
validation.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Claude Code (claude-sonnet-4-6)
Generated-by: Claude Code (claude-sonnet-4-6) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
> **Note**: A `{pr_number}.doc.rst` newsfragment will be added in a follow-up
> commit once this PR number is known, per the Airflow contribution
guidelines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
