amoghrajesh opened a new pull request, #75734:
URL: https://github.com/apache/airflow/pull/75734
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
<!--
If generative AI tooling has been used in the process of authoring this PR,
please
change below checkbox to `[X]` followed by the name of the tool, uncomment
the "Generated-by".
-->
- [x] Yes claude sonnet
<!--
Generated-by: [Tool Name] following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
### What
`KubernetesPodOperator` supports durable execution, but emits none of the
`resumable_job.*` counters that measure it. Those counters live in
`ResumableJobMixin`, and KPO achieves durability without using it (directly
using task state store) instead of the mixin, so it never reaches that code.
KPO is a large share of durable execution traffic by virtue of its
existence, so any answer to "how often did durable execution avoid redoing
work" is missing most of its real value today.
### Current behaviour
A `KubernetesPodOperator` with `durable=True` reconnects to a running pod
across retries and avoids resubmitting it, but emits nothing. In metrics, a
successful reattach and a fresh pod look identical, because neither produces a
counter.
### Proposed change
`get_or_create_pod` emits the existing `resumable_job.*` counters, tagged
`operator="KubernetesPodOperator"`. No new metric names.
KPO's pod outcomes map onto the mixin's branches one to one:
| KPO finds | Counters |
| -- | -- |
| nothing | `fresh_submit` |
| pod still running | `reconnect_attempt` + `reconnect_success` |
| pod already succeeded | `reconnect_attempt` + `already_succeeded` |
| pod failed or evicted | `reconnect_attempt` + `terminal_resubmit` |
Tags come from `ti.stats_tags` plus `operator`, matching what the mixin
emits.
### Changes of Note
**Reusing `resumable_job.*` over inventing a KPO specific name.** Easier
tracking when `operator` tag lets a reader count saves without caring which
mechanism produced them, which is the question being asked. The name is
slightly out of place for pods, but the counters are already live on 3.3, so a
rename would cost a deprecation cycle for nothing.
**A reconnect counts whether the pod came from the state store or from the
label search.** When the store read misses, KPO falls back to finding the pod
by label and then persists the identity it found. That is a real reattach, and
in access logs it
is indistinguishable from a fresh submit, because the trace is a miss
followed by a write. Metrics are not bound by that, so counting it as a
reconnect closes an undercount that cannot be closed on the log side.
**The phase decides the counter, not the cleanup.** `already_succeeded` and
`terminal_resubmit` are emitted from the pod phase, before
`process_pod_deletion` runs. Emitting after would make the counter depend on
`is_delete_operator_pod`, which is a cleanup setting rather than a resume
decision.
### User implications / backcompat
No breaking change, and no new configuration. Counters are only emitted when
`reattach_on_restart` is set, which is what `durable` resolves to, so
deployments that have turned durable execution off emit nothing.
Series cardinality grows for anyone already collecting `resumable_job.*`,
bounded by
`dag_id` x `task_id` x `operator`.
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
