Ewald-Hoffranzen opened a new issue, #62026:
URL: https://github.com/apache/airflow/issues/62026
### Apache Airflow version
3.1.7
### If "Other Airflow 3 version" selected, which one?
_No response_
### What happened?
### Apache Airflow version
3.1.7
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Task execution consistently fails with `Invalid auth token: Signature
verification failed` despite Scheduler and API-Server having identical JWT
secrets.
### What you think should happen instead?
Tasks should execute successfully with valid JWT authentication between
Scheduler and API-Server.
### How to reproduce
**Environment:**
- Airflow: 3.1.7 (Docker image: apache/airflow:3.1.7-python3.12)
- Executor: LocalExecutor
- Database: PostgreSQL 15
- Deployment: Docker Compose
**Configuration:**
```yaml
services:
airflow-scheduler:
environment:
AIRFLOW__CORE__INTERNAL_API_URL: "http://airflow-apiserver:8080";
AIRFLOW__CORE__EXECUTION_API_SERVER_URL:
"http://airflow-apiserver:8080/execution/";
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
AIRFLOW__CORE__EXECUTOR: "LocalExecutor"
airflow-apiserver:
environment:
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
```
**Steps:**
1. Deploy Airflow 3.1.7 with above configuration
2. Verify both services have identical JWT secrets: `airflow config
get-value secrets jwt_secret`
3. Trigger any DAG task
4. Observe immediate failure
### Operating System
Ubuntu 22.04 (Docker)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Docker Compose with separate containers for:
- airflow-apiserver
- airflow-scheduler
- airflow-dag-processor
- airflow-triggerer
- PostgreSQL (external, version 15)
### Anything else?
**Error Logs:**
Scheduler error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 966, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature
verification failed
Correlation-id=019c6702-76ee-7344-b98b-b3890d1e64df
```
API-Server error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py",
line 328, in avalidated_claims
self._verify_signature(signing_input, header, signature, key, algorithms)
jwt.exceptions.InvalidSignatureError: Signature verification failed
```
**Observations:**
1. Both services confirmed to have identical JWT secrets (86 bytes,
SHA512-compliant)
2. Problem persists after deleting jwt_secret from database and complete
service recreation
3. `InsecureKeyLengthWarning` appears intermittently, suggesting worker
subprocess uses cached/different secret
4. Setting `AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth`
is **ignored** by Task SDK
5. Worker subprocess in LocalExecutor creates tokens that API-Server cannot
validate
**Suspected Root Cause:**
Worker subprocess inherits or caches a different JWT secret than API-Server
uses for validation, possibly due to:
- Forked process not inheriting updated environment variables
- Task SDK hardcoded to use JWT regardless of auth_backends configuration
- Race condition in secret loading during worker subprocess creation
**Impact:**
Critical - Airflow 3.1.7 with LocalExecutor cannot execute any tasks.
**Workarounds Attempted:**
- ✗ Recreating all containers
- ✗ Deleting jwt_secret from database
- ✗ Setting Basic Auth (ignored by Task SDK)
- ✗ Ensuring identical secrets across all services
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
### What you think should happen instead?
_No response_
### How to reproduce
### Apache Airflow version
3.1.7
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Task execution consistently fails with `Invalid auth token: Signature
verification failed` despite Scheduler and API-Server having identical JWT
secrets.
### What you think should happen instead?
Tasks should execute successfully with valid JWT authentication between
Scheduler and API-Server.
### How to reproduce
**Environment:**
- Airflow: 3.1.7 (Docker image: apache/airflow:3.1.7-python3.12)
- Executor: LocalExecutor
- Database: PostgreSQL 15
- Deployment: Docker Compose
**Configuration:**
```yaml
services:
airflow-scheduler:
environment:
AIRFLOW__CORE__INTERNAL_API_URL: "http://airflow-apiserver:8080";
AIRFLOW__CORE__EXECUTION_API_SERVER_URL:
"http://airflow-apiserver:8080/execution/";
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
AIRFLOW__CORE__EXECUTOR: "LocalExecutor"
airflow-apiserver:
environment:
AIRFLOW__SECRETS__JWT_SECRET: "<86-byte-urlsafe-token>"
AIRFLOW__API__SECRET_KEY: "<86-byte-urlsafe-token>"
```
**Steps:**
1. Deploy Airflow 3.1.7 with above configuration
2. Verify both services have identical JWT secrets: `airflow config
get-value secrets jwt_secret`
3. Trigger any DAG task
4. Observe immediate failure
### Operating System
Ubuntu 22.04 (Docker)
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
Docker Compose with separate containers for:
- airflow-apiserver
- airflow-scheduler
- airflow-dag-processor
- airflow-triggerer
- PostgreSQL (external, version 15)
### Anything else?
**Error Logs:**
Scheduler error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py",
line 966, in _on_child_started
ti_context = self.client.task_instances.start(ti.id, self.pid, start_date)
airflow.sdk.api.client.ServerResponseError: Invalid auth token: Signature
verification failed
Correlation-id=019c6702-76ee-7344-b98b-b3890d1e64df
```
API-Server error:
```python
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/tokens.py",
line 328, in avalidated_claims
self._verify_signature(signing_input, header, signature, key, algorithms)
jwt.exceptions.InvalidSignatureError: Signature verification failed
```
**Observations:**
1. Both services confirmed to have identical JWT secrets (86 bytes,
SHA512-compliant)
2. Problem persists after deleting jwt_secret from database and complete
service recreation
3. `InsecureKeyLengthWarning` appears intermittently, suggesting worker
subprocess uses cached/different secret
4. Setting `AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth`
is **ignored** by Task SDK
5. Worker subprocess in LocalExecutor creates tokens that API-Server cannot
validate
**Suspected Root Cause:**
Worker subprocess inherits or caches a different JWT secret than API-Server
uses for validation, possibly due to:
- Forked process not inheriting updated environment variables
- Task SDK hardcoded to use JWT regardless of auth_backends configuration
- Race condition in secret loading during worker subprocess creation
**Impact:**
Critical - Airflow 3.1.7 with LocalExecutor cannot execute any tasks.
**Workarounds Attempted:**
- ✗ Recreating all containers
- ✗ Deleting jwt_secret from database
- ✗ Setting Basic Auth (ignored by Task SDK)
- ✗ Ensuring identical secrets across all services
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
### Operating System
ubuntu-20.04.3-live-server-amd64
### Versions of Apache Airflow Providers
_No response_
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
