[
https://issues.apache.org/jira/browse/AIRFLOW-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16776940#comment-16776940
]
Kiran Pathak commented on AIRFLOW-2167:
---------------------------------------
I like your solution. This bug is definitely a bit of frustrating :).
> Scheduler's clear_nonexistent_import_errors function should be called on
> first iteration
> ----------------------------------------------------------------------------------------
>
> Key: AIRFLOW-2167
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2167
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Affects Versions: 1.9.0
> Reporter: Casey
> Assignee: Casey
> Priority: Minor
> Attachments: Screen Shot 2018-03-02 at 2.08.29 PM.png
>
>
> In `airflow/jobs.py`, the `**clear_nonexistent_import_errors` function is not
> called until the amount of seconds defined by `dag_dir_list_interval` has
> elapsed. If the scheduler is not alive for the duration of
> `dag_dir_list_interval` (300 seconds) this cleanup never occurs. In some
> environments this could result in error messages displaying on the UI
> permanently, even if the DAG has been removed from the environment.
> It was previously an Airflow best practice to have the scheduler run N
> runtimes and terminate. Then, the scheduler would started again by an
> auxiliary process like Docker or Supervisor. This situation is what brought
> the bug to my attention.
> My suggested fix is to tweak jobs.py to run the import error cleanup on the
> first iteration and periodically as defined by `dag_dir_list_interval`. This
> way, a scheduler setup with a small number of runs will still have old errors
> cleaned up.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
