[
https://issues.apache.org/jira/browse/AIRFLOW-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786307#comment-16786307
]
Jeff Stein commented on AIRFLOW-972:
------------------------------------
We ran into this with some third party code (rollout.io's [rox
4.0.1|https://pypi.org/project/rox/]) that was using the multiprocessing
defaults (with the "fork" process start). They had a very short-lived process
that they were calling `terminate()` on.
Using something like this helped:
{code:java}
def signal_handler(signum, frame):
print("Ignoring SIGTERM from bad dependency")
sys.exit(0)
old_handler = signal.getsignal(signal.SIGTERM)
signal.signal(signal.SIGTERM, signal_handler)
# call third-party code that uses multiprocessing fork processes
...
signal.signal(signal.SIGTERM, old_handler){code}
> Airflow kills subprocesses created by task instances
> ----------------------------------------------------
>
> Key: AIRFLOW-972
> URL: https://issues.apache.org/jira/browse/AIRFLOW-972
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.7.1
> Reporter: Richard Moorhead
> Priority: Minor
>
> We have a task which creates multiple subprocesses via
> [joblib|https://pythonhosted.org/joblib/parallel.html]; we're noticing that
> airflow seems to kill the subprocesses prior to their completion. Is there
> any way around this behavior?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
