[
https://issues.apache.org/jira/browse/AIRFLOW-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126646#comment-17126646
]
ASF GitHub Bot commented on AIRFLOW-3607:
-----------------------------------------
mik-laj commented on pull request #7503:
URL: https://github.com/apache/airflow/pull/7503#issuecomment-639398221
We want to add the following test case:
- One DAG Run with many task instances in one pool.
- One DAG Run with many tasks instances with many pools
- Many DAGs Run with a limited number of tasks - below DAG concurrency limit
- Many DAGs Run with a limited number of tasks - above DAG concurrency limit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Decreasing scheduler delay between tasks
> ----------------------------------------
>
> Key: AIRFLOW-3607
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3607
> Project: Apache Airflow
> Issue Type: Improvement
> Components: scheduler
> Affects Versions: 1.10.0, 1.10.1, 1.10.2
> Environment: ubuntu 14.04
> Reporter: Amichai Horvitz
> Assignee: Amichai Horvitz
> Priority: Major
> Fix For: 2.0.0
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> I came across the TODO in airflow/ti_deps/deps/trigger_rule_dep (line 52)
> that says instead of checking the query for every task let the tasks report
> to the dagrun. I have a dag with many tasks and the delay between tasks can
> rise to 10 seconds or more, I already changed the configuration, added
> processes and memory, checked the code and did research, profiling and other
> experiments. I hope that this change will make a drastic change in the delay.
> I would be happy to discuss this solution, the research and other solutions
> for this issue.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
