Trying again. On Fri 23 Mar, 2018, 1:46 PM Amit Jain, <[email protected]> wrote:
> Hi, > > We have the use case where we have ETL jobs specific to single table per > dag and doing sanity checks of few of those tables according to a > particular business case in separate dag. > > One such business case demands us to have tables synced with entire day > business in our DWH. > > We want sanity check dag to be such that it verifies last run of ETL job > for concerned tables and make sure if last run didn't happen after 12 AM > then we need to trigger entire ETL job with retry granularity to ETL job's > operator, not as with complete subdag since our ETL jobs are costly. > > We would like to avoid using sensors as they will block our celery workers > since our ETL jobs are time-consuming as well. > > Please guide us to handle such use case. > > -- > Thanks, > Amit > > > >
