mjpieters opened a new issue #11894: URL: https://github.com/apache/airflow/issues/11894
Currently, the smart sensor group DAG creates smart sensors for all shards with a fixed poke interval, 180 seconds. This should really be configurable. Moreover, the sensor should find some kind of compromise between running at a fixed rate and honouring the original `poke_interval` values of the sensor operators it replaces. Now _all_ sensors managed by the smart sensor dag will be poked every 180 seconds, regardless. This could be handled like a cron scheduler instead: poke all the sensors that would have been poked since the last time the smart sensor ran, based on their `poke_interval` and `exponential_backoff` values. E.g. record and cache the next poke execution time per managed sensor, and only poke those whose next execution time has passed. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
