ai-projects-byte commented on issue #64476: URL: https://github.com/apache/airflow/issues/64476#issuecomment-4154938132
Two issues at play here — a misconfiguration and a real Airflow bug. **Root cause (misconfiguration):** The 401 Unauthorized on the execution API endpoint is caused by a JWT authentication failure between the worker subprocess and the API server. The warning in the logs — InsecureKeyLengthWarning: The HMAC key is 32 bytes long — indicates the secret in apiSecretKeySecretName is too short and/or not shared consistently across all components (scheduler, api-server, triggerer). In Airflow 3, all components must share a sufficiently long (≥ 64 bytes) identical key. Regenerating the secret with secrets.token_hex(64) and restarting all pods resolved this for us. **However, there IS a real Airflow bug here:** When the worker subprocess raises an httpx.HTTPStatusError (or any other unpicklable exception) and puts it into the multiprocessing.Queue, the scheduler crashes on deserialization with: ``` TypeError: HTTPStatusError.__init__() missing 2 required keyword-only arguments: 'request' and 'response' ``` This is because httpx.HTTPStatusError is not picklable. Airflow's LocalExecutor should wrap any exception in a picklable container (e.g., a string or a simple wrapper exception class) before putting it on the result queue. The current behavior turns any unpicklable worker-side exception into a full scheduler crash, which is clearly wrong — the scheduler should mark the task as failed and continue running, not crash entirely. The fix in local_executor.py should be somewhere around the result queue put/get path — either serialize exceptions to strings before enqueueing, or catch pickle errors on the get side and handle gracefully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
