Manikandan Ramakrishnan wrote:
> Had you set catchup parameter to True in your dag? If you had, it
> would have automatically run for the missing periods when it finally
> gets active.
That's a good point -- unfortunately, I had not. I considered updating
the DAG "catchup" parameter to toggle it to True, but I didn't want to
inadvertently kick off jobs for earlier missed data intervals from my
start date.
The way I ended up sort-of solving this was to put some placeholder
params for override data interval start and end timestamps, and then
add some logic in my operator to use those values if they are
non-empty, like this:
~~~
with DAG(
dag_id="some_dag",
catchup=False,
schedule_interval="@hourly",
params={
"override_interval_start": "",
"override_interval_end": "",
},
) as dag:
# etc ....
~~~
Then I can use the 'Trigger w/ Config' functionality, put timestamps
in those params, and it will use that as my effective "data interval".
This approach requires 'core.dag_run_conf_overrides_params' to be set
to True in the Airflow config.
Thanks,
Ben
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@airflow.apache.org
For additional commands, e-mail: users-h...@airflow.apache.org
