idrisakorede opened a new pull request, #61441:
URL: https://github.com/apache/airflow/pull/61441
# Fix #60269: Replace non-POSIX 'source' with '.' in EKS hook
## Problem
EksPodOperator fails with **401 Unauthorized** errors when running on
Debian/Ubuntu-based containers (Astronomer Runtime, official Airflow Docker
images, MWAA, etc.).
### Root Cause
The issue occurs in `airflow/providers/amazon/aws/hooks/eks.py` line 83,
where the `COMMAND` template uses `source`:
```bash
source {credentials_file}
```
**The problem:** `source` is a bash-specific builtin command, not a POSIX
standard command. On Debian/Ubuntu systems, `/bin/sh` is symlinked to `dash`
(not bash), which doesn't recognize `source`:
```bash
$ sh -c 'source /dev/null'
sh: 1: source: not found
```
This causes the credential loading to fail silently, resulting in 401
Unauthorized errors when the EKS token generation falls back to an empty
credential chain.
### Why This Is Hard to Detect
The bug is masked during local development when developers have
`~/.aws/credentials` mounted in containers:
1. `source {credentials_file}` fails silently (stderr not checked)
2. `eks_get_token.py` falls back to boto3's default credential chain
3. Finds credentials in `~/.aws/credentials` → token generation succeeds ✅
In production/cloud environments without `~/.aws/` directory, credentials
are only available via the temp file that failed to source, causing 401 errors ❌
## Solution
This PR implements two fixes:
### 1. Use POSIX-Compliant Dot Operator
Replace `source` with `.` (dot operator), which is POSIX-compliant and works
in all shells (bash, dash, sh):
```bash
# Before (bash-specific)
source {credentials_file}
# After (POSIX-compliant)
. {credentials_file}
```
### 2. Update Deprecated Kubernetes API Version
Update the authentication API version from deprecated `v1alpha1` to
`v1beta1`:
```python
# Before
AUTHENTICATION_API_VERSION = "client.authentication.k8s.io/v1alpha1"
# After
AUTHENTICATION_API_VERSION = "client.authentication.k8s.io/v1beta1"
```
**Note:** `v1alpha1` was deprecated in Kubernetes 1.24 and removed in 1.28.
## Changes Made
- Replace 'source' with POSIX-compliant '.' operator in COMMAND template
- Update deprecated v1alpha1 to v1beta1 Kubernetes API version
- Add shell compatibility tests for dash/POSIX shells
- Update comment to reflect POSIX compliance
## Testing
Added comprehensive test coverage in `test_eks.py`:
### New Test Classes
1. **TestEksHookShellCompatibility**
- `test_command_template_is_posix_compliant`: Verifies the template uses
`.` not `source`
- `test_credential_loading_works_with_dash`: Confirms credentials load
correctly with dash shell
- `test_source_command_fails_with_dash`: Documents the original bug
2. **TestEksHookKubernetesVersion**
- `test_uses_stable_kubernetes_api_version`: Ensures we're not using
deprecated v1alpha1
### Manual Testing
Verified in Breeze (Debian-based container):
```bash
$ ls -la /bin/sh
/bin/sh -> dash
$ sh -c '. /tmp/test.sh && echo $AWS_ACCESS_KEY_ID'
test_key # ✅ Works
$ sh -c 'source /tmp/test.sh && echo $AWS_ACCESS_KEY_ID'
sh: 1: source: not found # ❌ Fails
```
## Impact
This fix resolves 401 Unauthorized errors for all Debian/Ubuntu-based
Airflow deployments:
- ✅ Astronomer Runtime
- ✅ Official Apache Airflow Docker images
- ✅ Amazon MWAA (Managed Workflows for Apache Airflow)
- ✅ Any deployment where `/bin/sh` is `dash`
The change is backward-compatible as the `.` operator works in both bash and
dash shells.
---
Fixes #60269
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude AI was used for documentation)
<!--
Generated-by: Claude AI following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
