Pranaykarvi commented on PR #64485: URL: https://github.com/apache/airflow/pull/64485#issuecomment-4162467446
Thanks for raising this! From what I observed, this issue is tied specifically to `LocalExecutor` because it uses `multiprocessing.Queue` to send results (including exceptions) from worker subprocesses back to the scheduler. That means everything passed through needs to be pickleable and exceptions like `httpx.HTTPStatusError` aren’t, which leads to the crash. Other executors (like KubernetesExecutor or CeleryExecutor) don’t rely on multiprocessing queues in the same way they use external systems (like the K8s API, message brokers, or the metadata DB) to communicate task results, so they don’t run into this problem. I reproduced the issue using `LocalExecutor` and traced it through the multiprocessing flow, which is where the failure happens. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
