Ash Berlin-Taylor created AIRFLOW-5931:
------------------------------------------
Summary: Spawning new python interpreter is slow
Key: AIRFLOW-5931
URL: https://issues.apache.org/jira/browse/AIRFLOW-5931
Project: Apache Airflow
Issue Type: Improvement
Components: executors, worker
Affects Versions: 2.0.0
Reporter: Ash Berlin-Taylor
Assignee: Ash Berlin-Taylor
There are a number of places in the Executors and Task Runners where we spawn a
whole new python interpreter.
My profiling has shown that this is slow. Rather than running a fresh python
interpreter which then has to re-load all of Airflow and its dependencies we
should use {{os.fork}} when it is available/suitable which should speed up task
running, espeically for short lived tasks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
