Airflow isn't designed to work well with short schedule intervals. The guarantees that we give in terms of scheduling latency are limited as the platform isn't optimized for that specifically.
What is the type of operation that you are performing every 2 minutes? If you're doing data processing in microbatches you should look into data streaming solutions like Spark Streaming, Flink, Sanza, or Storm. Max On Mon, Mar 13, 2017 at 10:17 AM, Edgardo Vega <[email protected]> wrote: > Max, > > Sorry for so many ambiguous antecedents. > > I want to create a dag that does an operation waits 2 minutes and the runs > again over and over for all time. I don't know if that is possible to do > with airflow or somehow trick airflow into doing this. > > I hope that clears things up. > > Cheers, > > Edgardo > > > On Mon, Mar 13, 2017 at 12:39 PM, Maxime Beauchemin < > [email protected]> wrote: > > > "Would this be possible in airflow?" > > > > What do you mean by "this"? > > > > Max > > > > On Mon, Mar 13, 2017 at 9:09 AM, Edgardo Vega <[email protected]> > > wrote: > > > > > We are currently trying to port our current solution into airflow. What > > is > > > currently stumping me is we have a few tasks we have running pretty > much > > > all the time. Once it is done we wait a few minutes and kick off > another. > > > > > > Would this be possible in airflow? > > > > > > -- > > > Cheers, > > > > > > Edgardo > > > > > > > > > -- > Cheers, > > Edgardo >
