I wholeheartedly agree with TP on that one.  I think while some time
ago "data date" could make sense, Airflow's future is much more than just
processing data intervals.
This is the primary use case and this is where Airflow shines od course,
but one of the good examples of how Airflow is used out there, and while we
are not really encouraging it, there are not only legitimate, but also
something that I hope Airflow will treat as first-time citizens soon (and
it kind of already is with custom timetables).

Just an example here - for me one of the most eye-opening talks in last
year's Airflow Summit
https://airflowsummit.org/sessions/2021/provision-as-a-service/
In this talk Cloudflare engineers explain how they manage the CloudFlare
infrastructure using Airflow.

The "Data date" has no meaning in this case. But the "logical Date" (which
is the vaguest-possible one as TP explained) continues to have one. This is
the "logical date of the infrastructure provisioning". Thanks to Airflow
(as I understand it) Cloudflare is able to re-provision their services
to "yesterday's logical date infrastructure"  today - for example.

That would not fly with "data date".

J,

Reply via email to