symat opened a new pull request, #77163:
URL: https://github.com/apache/airflow/pull/77163

   `EksPodOperator(deferrable=True)` tasks fail when several of them defer at 
about the same time. The trigger dies before it watches the pod, and the task 
fails with `TaskDeferralError: Trigger failure`:
   
   ```
   File ".../airflow/providers/amazon/aws/triggers/eks.py", line 203, in run
       with eks_hook.generate_config_file(
   File ".../airflow/providers/amazon/aws/hooks/eks.py", line 646, in 
generate_config_file
       del os.environ["AWS_STS_REGIONAL_ENDPOINTS"]
   KeyError: 'AWS_STS_REGIONAL_ENDPOINTS'
   ```
   
   `EksHook.generate_config_file` set `AWS_STS_REGIONAL_ENDPOINTS=regional` in 
the process environment around the STS endpoint lookup and deleted it 
afterwards. All triggers of a triggerer share that environment. The lookup 
creates an `StsHook`, whose connection lookup is a sync SDK call that the 
triggerer serves through the event loop, so other triggers run while it waits. 
When a second `EksPodTrigger` starts in that window, the first one to finish 
deletes the variable and the other one's `del` raises `KeyError`. Starts bunch 
up on bursts, on retries and when a triggerer restarts and its triggers move to 
another instance, so retries tend to fail the same way. Independently of 
concurrency, the hook also deleted a value the user had set themselves, for the 
whole process.
   
   botocore resolves the regional STS endpoint by default (since 1.40; the 
provider requires `botocore>=1.43.6`), and `AwsEksExecutor` already builds its 
STS URL without touching the environment. The hook now does the same.
   
   Behaviour change: a deployment that explicitly sets 
`AWS_STS_REGIONAL_ENDPOINTS=legacy` now gets the global STS endpoint in the 
generated kubeconfig, instead of the regional one the hook forced before. This 
matches `AwsEksExecutor`.
   
   Tested on Airflow 3.3.2 with `apache-airflow-providers-amazon` 9.36.0, two 
triggerers and 100 Dag runs started 0.5 s apart, each running one deferrable 
`EksPodOperator` task against a real EKS cluster:
   
   - Without the change, 55 of the 100 tasks failed all their tries with the 
`KeyError` above.
   - With the change applied to 9.36.0, three runs of the same test had no 
trigger failures and no `KeyError`.
   
   The new unit test sets `AWS_STS_REGIONAL_ENDPOINTS=legacy` and checks that 
the kubeconfig follows it and that the variable is still set afterwards. It 
fails on `main` and passes with this change.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — Devin (Fusion)
   
   Generated-by: Devin (Fusion) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to