amoghrajesh opened a new pull request, #70298:
URL: https://github.com/apache/airflow/pull/70298
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [x] Yes - claude sonnet 4.6
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
---
### Important Note
**What "durable" and "deferrable" mean in the PR:** there are two different
reasons an operator can survive a crash, and they work in totally different
ways. "Durable" means the operator own code remembers where its job was and
reconnects to it instead of starting over, and this operates using task state
store in **synchronous mode**. "Deferrable" is running using a trigger, in
**asynchronous mode**. The end result looks similar to a user (nothing gets
redone), but one is a promise the operator has to keep and the other is just
how the system already works for everyone.
### What
Follow up to the merged `supports_durable_execution` badge PR (#67236). Now
that we have badget support, we should also have a few filters for ease of
navigation for two related-but-distinct capabilities: "durable" (anything that
survives a crash without redoing work -- `ResumableJobMixin`/manual-marker
operators, or deferrable ones) and "deferrable" (the narrower, deferrable-only
subset). The registry had no deferrable detection at all before this change.
### Current behaviour
The Registry only tracks `supports_durable_execution` (`ResumableJobMixin`
or the manual `__supports_durable_execution` marker coming in PR:
https://github.com/apache/airflow/pull/70291). Deferrable operators -- a much
more common capability across most cloud providers -- have no signal anywhere
in the pipeline, and there's no way to filter a provider's module list by
either capability.
### Proposed change
- **Detection**: `supports_deferrable(cls)` added to
`dev/registry/extract_parameters.py` -- checks for a `deferrable` constructor
parameter, reusing the existing `get_params_from_class()` machinery already
used for `parameters.json` (no new introspection needed).
- **Schema**: `supports_deferrable: bool = False` added to `ModuleContract`,
wired into `make_entry()` alongside `supports_durable_execution` (14 fields
total, kept as an independent signal, not merged into the existing one).
- **Frontend**: two independent badges on module cards ("Durable",
"Deferrable": teal/blue) shown per their own signal, since an operator can have
either, both, or neither. Two filter toggles on a provider's module list:
"Durable (includes deferrable)" and "Deferrable only".
### Changes of Note
- **The two filters are a deliberate union/subset pair, not independent
booleans.** "Durable" is `supports_durable_execution OR supports_deferrable` --
checking it shows every deferrable operator plus every
`ResumableJobMixin`/manual-marker operator, even ones that aren't deferrable.
"Deferrable only" is `supports_deferrable` alone, never OR'd. This means an
operator that is both durable and deferrable appears under *both* filters --
verified this is intentional and not confusing in practice, since the two
independent badges on the card explain why.
- The checkbox labels ("Durable (includes deferrable)" / "Deferrable only")
were chosen specifically to make that asymmetric containment relationship
(deferrable `⊆` durable, never the reverse) visible at the point of clicking,
rather than relying on a tooltip or a caption someone could miss.
- `is_deferrable` inheriting a parent's `__init__` unmodified still counts
as deferrable (unlike the `supports_durable_execution` manual-marker case,
which deliberately does *not* propagate to subclasses) -- inheriting an
unmodified constructor is a real functional guarantee the parameter still
works, whereas inheriting a durable-execution *claim* says nothing about
whether an overridden `execute()` still preserves it.
### Testing
- Built breeze to extract the provider data using: `breeze registry
extract-data`
- Built the actual Eleventy site against this real data and confirmed in the
rendered HTML: both badges appear on the right operator cards (e.g. Databricks'
page shows both a "Durable" and "Deferrable" badge on
`DatabricksRunNowOperator`), both filter checkboxes render with the intended
labels, and the `data-durable`/`data-deferrable` attributes are correctly
populated per the OR/non-OR logic on every module checked.
For current changes, this is the behaviour:
- **Databricks**: `DatabricksSubmitRunOperator` and
`DatabricksRunNowOperator` are both durable *and* deferrable.
`DatabricksNotebookOperator`, `DatabricksSQLStatementsOperator`,
`DatabricksTaskBaseOperator`, `DatabricksTaskOperator`, and
`DatabricksSQLStatementsSensor` are deferrable only. The rest of the provider's
operators/hooks/sensors are neither.
- **Apache Spark**: `SparkSubmitOperator` is durable but *not* deferrable
(no `deferrable` param) -- this is the single durable-only case across the
entire registry.
- **Google/BigQuery**: `BigQueryInsertJobOperator` is both durable and
deferrable, same pattern as Databricks.
- **Amazon/Redshift**: `RedshiftDataOperator` is both durable and deferrable.
- **Snowflake**: `SnowflakeSqlApiOperator` is both durable and deferrable.
Across the full dataset: 1779 modules total, 6 durable, 217 deferrable, 5
with both (the five listed above).
Some screenshots:
In this image, "Durable (includes deferrable)" is selected, and you can see
both durable + deferrable (BigQueryInsertJobOperator) and just deferrable ones
selected
<img width="1007" height="717" alt="image"
src="https://github.com/user-attachments/assets/b8313ece-8f0c-4c69-8057-459c6eed57f0";
/>
"Deferrable only selection":
<img width="1007" height="717" alt="image"
src="https://github.com/user-attachments/assets/f24eea5e-0eb0-4255-af3e-a7d6dcc8db76";
/>
Operator like snowflake has only one operator with both deferrable and
durable property, so its here:
<img width="1007" height="284" alt="image"
src="https://github.com/user-attachments/assets/9034c069-557c-4ae2-a9e1-305ce0a9784f";
/>
Spark is one of few operators with only "durable", and no deferrable option:
<img width="1007" height="284" alt="image"
src="https://github.com/user-attachments/assets/dbf6281f-23f3-449c-9800-0c19ad6be1e5";
/>
### What's next
Interactive click-through verification (actually toggling the checkboxes in
a running browser to confirm `filterModules()` behaves correctly) hasn't been
done -- only the generated HTML/data have been checked, not runtime JS
behavior. Registry-wide (cross-provider) discovery -- "which are all the
durable/deferrable operators across the whole registry," via a Pagefind search
filter facet -- was discussed and deliberately deferred to a separate follow-up.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
