symat opened a new pull request, #77163:
URL: https://github.com/apache/airflow/pull/77163
`EksPodOperator(deferrable=True)` tasks fail when several of them defer at
about the same time. The trigger dies before it watches the pod, and the task
fails with `TaskDeferralError: Trigger failure`:
```
File ".../airflow/providers/amazon/aws/triggers/eks.py", line 203, in run
with eks_hook.generate_config_file(
File ".../airflow/providers/amazon/aws/hooks/eks.py", line 646, in
generate_config_file
del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
KeyError: 'AWS_STS_REGIONAL_ENDPOINTS'
```
`EksHook.generate_config_file` set `AWS_STS_REGIONAL_ENDPOINTS=regional` in
the process environment around the STS endpoint lookup and deleted it
afterwards. All triggers of a triggerer share that environment. The lookup
creates an `StsHook`, whose connection lookup is a sync SDK call that the
triggerer serves through the event loop, so other triggers run while it waits.
When a second `EksPodTrigger` starts in that window, the first one to finish
deletes the variable and the other one's `del` raises `KeyError`. Starts bunch
up on bursts, on retries and when a triggerer restarts and its triggers move to
another instance, so retries tend to fail the same way. Independently of
concurrency, the hook also deleted a value the user had set themselves, for the
whole process.
botocore resolves the regional STS endpoint by default (since 1.40; the
provider requires `botocore>=1.43.6`), and `AwsEksExecutor` already builds its
STS URL without touching the environment. The hook now does the same.
Behaviour change: a deployment that explicitly sets
`AWS_STS_REGIONAL_ENDPOINTS=legacy` now gets the global STS endpoint in the
generated kubeconfig, instead of the regional one the hook forced before. This
matches `AwsEksExecutor`.
Tested on Airflow 3.3.2 with `apache-airflow-providers-amazon` 9.36.0, two
triggerers and 100 Dag runs started 0.5 s apart, each running one deferrable
`EksPodOperator` task against a real EKS cluster:
- Without the change, 55 of the 100 tasks failed all their tries with the
`KeyError` above.
- With the change applied to 9.36.0, three runs of the same test had no
trigger failures and no `KeyError`.
The new unit test sets `AWS_STS_REGIONAL_ENDPOINTS=legacy` and checks that
the kubeconfig follows it and that the variable is still set afterwards. It
fails on `main` and passes with this change.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Devin (Fusion)
Generated-by: Devin (Fusion) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]