[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820264#comment-16820264 ] Ignas commented on AIRFLOW-4339: I am running LocalExecutor and they are picked by the backfill command, the main problem was that when tasks are cleared the backfill will pick all the DagRuns at once, ignoring the max_active_runs parameter. The workaround with --reset_dagruns, --ignore_dependencies and --task-regex is ok for me, this way I do not need to clear the tasks. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820258#comment-16820258 ] Tao Feng commented on AIRFLOW-4339: --- [~Ignas] I am in a conference, could take a look in the night, but all the clear tasks should be picked by scheduler. Which executor do you use for backfill? The patch has been running in our prod without any issues so far. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820207#comment-16820207 ] Ignas commented on AIRFLOW-4339: Together with --reset_dagruns? I do use task-regex, the problem is that after clearing the tasks I have a few hundreds of "running" DagRuns, and if I then run backfill command it will start executing all of it which completely hangs the whole server > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820201#comment-16820201
]
Ash Berlin-Taylor commented on AIRFLOW-4339:
In which case a combo of {{--ignore_dependencies}} and {{--task-regex}}
> Cleared backfill tasks do not get picked by the scheduler
> -
>
> Key: AIRFLOW-4339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4339
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill, scheduler
>Affects Versions: 1.10.1
>Reporter: Ignas
>Priority: Blocker
>
> When a task in a successful backfill run is cleared DagRun with running state
> is created but it is not picked by the scheduler.
> If there are many of such DagRuns and the backfill command is executed, all
> the DagRuns will be ran at the same time, not following the max_active_runs
> parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820199#comment-16820199 ] Ignas commented on AIRFLOW-4339: --reset_dagruns will backfill all the tasks in the dag, in my case I only need to backfill one of the tasks. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[
https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820191#comment-16820191
]
Ash Berlin-Taylor commented on AIRFLOW-4339:
There's a {{--reset_dagruns}} option to backfil that will help it sounds like.
Arguably a bug that clearing in the UI changes the state of backfilled runs
> Cleared backfill tasks do not get picked by the scheduler
> -
>
> Key: AIRFLOW-4339
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4339
> Project: Apache Airflow
> Issue Type: Bug
> Components: backfill, scheduler
>Affects Versions: 1.10.1
>Reporter: Ignas
>Priority: Blocker
>
> When a task in a successful backfill run is cleared DagRun with running state
> is created but it is not picked by the scheduler.
> If there are many of such DagRuns and the backfill command is executed, all
> the DagRuns will be ran at the same time, not following the max_active_runs
> parameter.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4339) Cleared backfill tasks do not get picked by the scheduler
[ https://issues.apache.org/jira/browse/AIRFLOW-4339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820179#comment-16820179 ] Ignas commented on AIRFLOW-4339: But then why are the DagRuns created after clearing the tasks. Wouldn't it be better if they were created by the backfill process, so that the task/dagrun limits are taken into account? Currently if a lot of backfilled tasks need to be rerun, the only way is to clear it, then mark the created dagruns as successful, and then run the backfill which will create dagruns one by one, not all at the same time. > Cleared backfill tasks do not get picked by the scheduler > - > > Key: AIRFLOW-4339 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4339 > Project: Apache Airflow > Issue Type: Bug > Components: backfill, scheduler >Affects Versions: 1.10.1 >Reporter: Ignas >Priority: Blocker > > When a task in a successful backfill run is cleared DagRun with running state > is created but it is not picked by the scheduler. > If there are many of such DagRuns and the backfill command is executed, all > the DagRuns will be ran at the same time, not following the max_active_runs > parameter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
