Vamsi-klu opened a new pull request, #63610: URL: https://github.com/apache/airflow/pull/63610
## Summary Pre-create cache directories used by exec-based authentication plugins before establishing the Kubernetes API connection in `KubernetesHook.get_conn()`. When multiple tasks run concurrently on the same Celery worker, each may invoke an exec-based credential plugin (e.g., AWS IAM authenticator) in parallel. Some plugin implementations create their cache directories with `os.makedirs()` without `exist_ok=True`, which causes a `FileExistsError` race condition when two processes attempt the creation simultaneously. This fix defensively pre-creates known cache directories so they already exist by the time any exec plugin runs. closes: #60943 ## Test plan - [x] `test_creates_cache_dirs` — verifies directories are created when missing - [x] `test_no_error_when_dir_already_exists` — verifies no error when dirs already exist - [x] `test_get_conn_calls_ensure_cache_dirs` — verifies `get_conn()` calls the pre-creation function - [x] Ruff lint and format pass - [x] All tests pass locally > **Note:** This fix was originally included in PR #62656 (structlog serialization fix) by mistake. > It has been split out into its own PR as requested by reviewers. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Claude Opus 4.6) Generated-by: Claude Code (Claude Opus 4.6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
