[
https://issues.apache.org/jira/browse/AIRFLOW-1689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Angela Zhang updated AIRFLOW-1689:
----------------------------------
Description:
This happens even though we haven't reached the max active run / concurrency.
see:
!https://d3uik7083b30dy.cloudfront.net/items/043p0z0h2h0J42393E3i/Image%202017-10-06%20at%2011.26.35%20AM.png?X-CloudApp-Visitor-Id=1681354&v=3020d646!
This happens to multiple of our dags, where if previous dag runs of this dag
have too many failed / running tasks (even though it's still lower than the
concurrency / max active dags runs, concurrency=16, max_active_runs=16), the
scheduler does not schedule any more tasks in the latest dag run for this dag.
!https://d3uik7083b30dy.cloudfront.net/items/0r3r2B141M233Z2r0C1B/Image%202017-10-06%20at%2011.38.17%20AM.png?X-CloudApp-Visitor-Id=1681354&v=1903b2e6!
We can make the scheduler schedule more tasks if we mark the previous dag runs
as successful, but this still seems like a bug in airflow.
was:
see:
!https://d3uik7083b30dy.cloudfront.net/items/043p0z0h2h0J42393E3i/Image%202017-10-06%20at%2011.26.35%20AM.png?X-CloudApp-Visitor-Id=1681354&v=3020d646!
This happens to multiple of our dags, where if previous dag runs of this dag
have too many failed / running tasks (even though it's still lower than the
concurrency / max active dags runs, concurrency=16, max_active_runs=16), the
scheduler does not schedule any more tasks in the latest dag run for this dag.
!https://d3uik7083b30dy.cloudfront.net/items/0r3r2B141M233Z2r0C1B/Image%202017-10-06%20at%2011.38.17%20AM.png?X-CloudApp-Visitor-Id=1681354&v=1903b2e6!
We can make the scheduler schedule more tasks if we mark the previous dag runs
as successful, but this still seems like a bug in airflow.
> Airflow doesn't schedule tasks if there are too many failed tasks in a
> previous dag run
> ---------------------------------------------------------------------------------------
>
> Key: AIRFLOW-1689
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1689
> Project: Apache Airflow
> Issue Type: Bug
> Reporter: Angela Zhang
>
> This happens even though we haven't reached the max active run / concurrency.
> see:
> !https://d3uik7083b30dy.cloudfront.net/items/043p0z0h2h0J42393E3i/Image%202017-10-06%20at%2011.26.35%20AM.png?X-CloudApp-Visitor-Id=1681354&v=3020d646!
> This happens to multiple of our dags, where if previous dag runs of this dag
> have too many failed / running tasks (even though it's still lower than the
> concurrency / max active dags runs, concurrency=16, max_active_runs=16), the
> scheduler does not schedule any more tasks in the latest dag run for this
> dag.
> !https://d3uik7083b30dy.cloudfront.net/items/0r3r2B141M233Z2r0C1B/Image%202017-10-06%20at%2011.38.17%20AM.png?X-CloudApp-Visitor-Id=1681354&v=1903b2e6!
> We can make the scheduler schedule more tasks if we mark the previous dag
> runs as successful, but this still seems like a bug in airflow.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
