[
https://issues.apache.org/jira/browse/AIRFLOW-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16608732#comment-16608732
]
kkkkk commented on AIRFLOW-2895:
--------------------------------
The logs being spammed is not the only problem of this bug. It is also causing
a very high CPU load, which is particularly problematic on lower-end servers.
Even on my local development machine it causes the CPU to use up the entire
core I have given it.
Perhaps it would be worth considering releasing another airflow update that
includes a fix for this sooner rather than later.
> Prevent scheduler from spamming heartbeats/logs
> -----------------------------------------------
>
> Key: AIRFLOW-2895
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2895
> Project: Apache Airflow
> Issue Type: Bug
> Components: scheduler
> Reporter: Dan Davydov
> Assignee: Dan Davydov
> Priority: Major
>
> There seems to be a couple of problems with
> [https://github.com/apache/incubator-airflow/pull/2986] that cause the sleep
> to not trigger and Scheduler heartbeating/logs to be spammed:
> # If all of the files are being processed in the queue, there is no sleep
> (can be fixed by sleeping for min_sleep even if there are no files)
> # I have heard reports that some files can return a parsing time that is
> monotonically increasing for some reason (e.g. file actually parses in 1s
> each loop, but the reported duration seems to use the very time the file was
> parsed as the start time instead of the last time), I haven't confirmed this
> but it sounds problematic.
> To unblock the release I'm reverting this PR for now. It should be re-added
> with tests/mocking.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
