yrqls21 commented on issue #2986: [AIRFLOW-2027] Only trigger sleep in scheduler after all files have parsed URL: https://github.com/apache/incubator-airflow/pull/2986#issuecomment-412637527 @aoen Agree on sleep for a minimal time interval, it should not be a huge problem to sleep at least one second every loop( have that in my branch for a while and looks good until now). I'm not so sure about monotonically increasing, what I can see is that if we have parse files faster than we set file path, then if file was removed/have import error and not updating the last_finish_time in the 2nd parsing round, we'll still use the last_finish_time from the 1st round and get some non-desirable duration value.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
