ltshb commented on issue #50188:
URL: https://github.com/apache/arrow/issues/50188#issuecomment-6981019921
I'm seeing the same hang with pyarrow 26.0.1.
Setup: Airflow 3.4, Python 3.14, Airflow on EKS. Each task runs in its own
EKS pod and gets AWS credentials through IRSA (web identity), which matches the
STS/web-identity trigger mentioned above.
The task finishes its work, but the process never exits. Downgrading to
pyarrow 23.0.1 fixes it.
Workaround for Airflow if you want to stay on a recent pyarrow: hard-exit
the task process in an `on_success_callback`. In Airflow 3 the task has already
reported SUCCESS when this callback runs, and exit code 0 keeps that state, so
the deadlocking atexit handler is skipped:
```python
import os
import sys
def hard_exit_on_success(context):
# Workaround for https://github.com/apache/arrow/issues/50188
sys.stdout.flush()
sys.stderr.flush()
os._exit(0) # must be 0, otherwise Airflow marks the task as failed
@dag(default_args={"on_success_callback": hard_exit_on_success}, ...)
def my_dag(): ...
```
Caveat: it must be the last success callback, because anything after it
(other callbacks, success listeners) won't run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]