[
https://issues.apache.org/jira/browse/AIRFLOW-6361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046757#comment-17046757
]
ASF GitHub Bot commented on AIRFLOW-6361:
-----------------------------------------
stale[bot] commented on pull request #6905: [AIRFLOW-6361] Run LocalTaskJob
directly in Celery task
URL: https://github.com/apache/airflow/pull/6905
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Run LocalTaskJob directly in Celery task
> ----------------------------------------
>
> Key: AIRFLOW-6361
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6361
> Project: Apache Airflow
> Issue Type: Improvement
> Components: executors
> Affects Versions: 1.10.6
> Reporter: Kamil Bregula
> Priority: Major
> Labels: performance
>
> Hello,
> Celery runs the CLI first command, which contains LocalTaskJob. LocalTaskJob
> is responsible for starting the next user-code process. This level of
> isolation is redundant because LocalTaskJob doesn't execute unsafe code. The
> first command is run by a new process creation, not by a fork, so this is an
> expensive operation.
> According to preliminary measurements, this change results in an increase in
> performance close to 30%.
> I will provide more information in PR.
> Best regards
> Kamil Bregula
> After:
> ```
> real 0m38.394s
> user 0m4.340s
> sys 0m1.600s
> real 0m38.355s
> user 0m4.700s
> sys 0m1.340s
> real 0m38.675s
> user 0m4.760s
> sys 0m1.530s
> real 0m38.488s
> user 0m4.770s
> sys 0m1.280s
> real 0m38.434s
> user 0m4.600s
> sys 0m1.390s
> real 0m38.378s
> user 0m4.500s
> sys 0m1.270s
> real 0m38.106s
> user 0m4.200s
> sys 0m1.100s
> real 0m38.082s
> user 0m4.170s
> sys 0m1.030s
> real 0m38.173s
> user 0m4.290s
> sys 0m1.340s
> real 0m38.161s
> user 0m4.460s
> sys 0m1.370s
> ```
> Before:
> ```
> real 0m53.488s
> user 0m5.140s
> sys 0m1.700s
> real 1m8.288s
> user 0m6.430s
> sys 0m2.200s
> real 0m53.371s
> user 0m5.330s
> sys 0m1.630s
> real 0m58.939s
> user 0m6.470s
> sys 0m1.730s
> real 0m53.255s
> user 0m4.950s
> sys 0m1.640s
> real 0m58.802s
> user 0m5.970s
> sys 0m1.790s
> real 0m58.449s
> user 0m5.380s
> sys 0m1.580s
> real 0m53.308s
> user 0m5.120s
> sys 0m1.430s
> real 0m53.485s
> user 0m5.220s
> sys 0m1.290s
> real 0m53.387s
> user 0m5.020s
> sys 0m1.590s
> ```
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
