feluelle commented on issue #6428: [AIRFLOW-5753] Add DailyLatestOnlyOperator URL: https://github.com/apache/airflow/pull/6428#issuecomment-546606824 > This is useful when we scrape external data from third-parties on an hourly basis/for multiple times in a day to keep data up-to-date. If the DAG is paused for several days, we want to avoid multiple runs in a day, but still want to ensure data is filled in every daily window. We (at work) have a similiar case where we are always requesting data from the last 3 months. Our first task is a `ShortCircuitOperator` that checks if the current run really needs to run i.e. the execution date is in a list of dates or it is the latest execution date.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
