The GitHub Actions job "Tests" on airflow.git/fix/kpo-aws-cli-cache-race-condition has failed. Run started by GitHub user Arunodoy18 (triggered by Arunodoy18).
Head commit for run: 17bf3c26000121414ef712c89b1ea108ef17112c / Arunodoy18 <[email protected]> Fix AWS CLI cache race condition in KubernetesPodOperator When multiple KubernetesPodOperator tasks run in parallel on the same Celery worker with EKS authentication, they concurrently invoke the AWS CLI for token generation. This causes a race condition where all tasks attempt to create ~/.aws/cli/cache simultaneously, resulting in FileExistsError and task failure before pod creation. This commit adds AWS CLI cache directory isolation per task execution: - Created _isolated_aws_cli_cache() context manager in kubernetes hook - Sets task-specific AWS_SHARED_CREDENTIALS_FILE env var with UUID - Wraps all config.load_kube_config() calls with the isolation context - Ensures cleanup of temporary directories after authentication - Added concurrent authentication test to verify no race conditions The fix is transparent to users - no configuration changes required. Non-EKS and in-cluster setups are unaffected. Each parallel KPO task now uses its own isolated AWS cache directory, preventing filesystem collisions during concurrent EKS authentication. Fixes race condition causing FileExistsError in parallel KPO execution. Report URL: https://github.com/apache/airflow/actions/runs/21328107844 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
