Arunodoy18 opened a new pull request, #61025: URL: https://github.com/apache/airflow/pull/61025
When running multiple KubernetesPodOperator tasks in parallel on the same Celery worker, Kubernetes authentication against Amazon EKS may intermittently fail with a FileExistsError originating from the AWS CLI cache directory. This happens because concurrent executions of aws eks get-token attempt to create a shared cache directory without synchronization. This PR prevents the collision by isolating the AWS CLI cache location per task execution during Kubernetes authentication. Problem: KubernetesPodOperator authenticates to the Kubernetes API server on the Airflow worker before pod creation In EKS environments, authentication relies on aws eks get-token The AWS CLI uses a shared cache directory (~/.aws/cli/cache) and is not concurrency-safe Parallel KPO tasks on the same worker can race while creating this directory, causing authentication to fail The failure occurs before pod creation and may surface as a Kubernetes API authentication error. Solution: The Kubernetes authentication path used by KubernetesPodOperator is updated to use a task-scoped AWS CLI cache directory. This avoids filesystem contention when multiple tasks authenticate concurrently on the same worker. The change: Does not modify AWS CLI behavior Does not affect task scheduling or execution semantics Is limited to the Kubernetes authentication path Tests: Added coverage to ensure Kubernetes authentication does not fail under concurrent execution due to shared filesystem state Backward Compatibility No breaking changes No impact on non-EKS or non-AWS Kubernetes configurations Existing DAGs continue to function as before, with improved reliability under parallel execution Related Issue Closes #60943 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
