I have a few DAGs that use time sensors to wait until data is ready, which can be several days.
I have one daily DAG where, for each execution date, I have to repull the data for the next 7 days to capture changes (late arriving revenue data). This DAG currently starts 7 TimeDeltaSensors for each execution days with delays that range from 0 to 6 days. I was wondering what the recommendation is for cases like this where a large number of sensors is needed. Are there ways to reduce the footprint of these sensors so that they use less CPU and memory? I noticed that in one of the DAGs that Germain Tanguy had in the presentation he shared today a sensor was set to time out every 30 seconds but had a large retry count so instead of running constantly, it runs every 15 minutes for 30 seconds and then dies. Are other people using this pattern? Do you have other suggestions? Thanks, Pedro
