Pranaykarvi opened a new pull request, #64485:
URL: https://github.com/apache/airflow/pull/64485
Closes #64476
When LocalExecutor runs a task in a subprocess, exceptions are passed back
via multiprocessing.Queue using pickle. Some exceptions like
httpx.HTTPStatusError are not pickle-safe, causing:
TypeError: HTTPStatusError.__init__() missing 2 required keyword-only
arguments: 'request' and 'response'
This crashes the scheduler loop and takes down the entire scheduler pod.
## Root Cause
Raw exception objects were placed directly onto the result queue:
output.put((key, TaskInstanceState.FAILED, e))
## Fix
Wrap the exception in a plain Exception before queuing:
safe_exc = Exception(f"{type(e).__name__}:
{str(e)}\n{traceback.format_exc()}")
output.put((key, TaskInstanceState.FAILED, safe_exc))
Applied to both ExecuteTask and ExecuteCallback branches.
## Impact
- Scheduler no longer crashes on non-picklable exceptions
- Full debugging info preserved
- Minimal change, no other executor behaviour affected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
