[
https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17209522#comment-17209522
]
ASF GitHub Bot commented on AIRFLOW-3607:
-----------------------------------------
yuqian90 commented on pull request #11010:
URL: https://github.com/apache/airflow/pull/11010#issuecomment-704918738
> As mentioned in [#11119
(comment)](https://github.com/apache/airflow/pull/11119#issuecomment-702372954)
we are no more accepting new features to 1-10 branch :< I'm sorry. The 2.0
alpha should be released soon 🎉
Hi, @turbaszek and @dimberman I agree that #11119 can be closed since
it's a brand new feature. However this one #11010 is not a new feature. It's
more of a fix to 1.10.*. When DAGs are just a little bit larger than usual (500
tasks) and when user tries to run dozens of DagRuns in at the same time (around
30), airflow-scheduler becomes super slow. This PR is the fix for that
backported for 1.10.*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Decreasing scheduler delay between tasks
> ----------------------------------------
>
> Key: AIRFLOW-3607
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 1.10.0, 1.10.1, 1.10.2
> Environment: ubuntu 14.04
> Reporter: Amichai Horvitz
> Assignee: Amichai Horvitz
> Priority: Major
> Fix For: 2.0.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52)
> that says instead of checking the query for every task let the tasks report
> to the dagrun. I have a dag with many tasks and the delay between tasks can
> rise to 10 seconds or more, I already changed the configuration, added
> processes and memory, checked the code and did research, profiling and other
> experiments. I hope that this change will make a drastic change in the delay.
> I would be happy to discuss this solution, the research and other solutions
> for this issue. Â
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
