kaxil commented on code in PR #70167:
URL: https://github.com/apache/airflow/pull/70167#discussion_r3683711961
##########
airflow-core/docs/core-concepts/dag-run.rst:
##########
@@ -138,20 +151,33 @@ then you will want to turn catchup off, which is the
default setting or can be d
)
In the example above, if the Dag is picked up by the scheduler daemon on
-2016-01-02 at 6 AM, (or from the command line), a single Dag Run will be
created
-with a data between 2016-01-01 and 2016-01-02, and the next one will be created
-just after midnight on the morning of 2016-01-03 with a data interval between
-2016-01-02 and 2016-01-03.
-
-Be aware that using a ``datetime.timedelta`` object as schedule can lead to a
different behavior.
-In such a case, the single Dag Run created will cover data between 2016-01-01
06:00 and
-2016-01-02 06:00 (one schedule interval ending now). For a more detailed
description of the
-differences between a cron and a delta based schedule, take a look at the
-:ref:`timetables comparison <Differences between the cron and delta data
interval timetables>`
-
-If the ``dag.catchup`` value had been ``True`` instead, the scheduler would
have created a Dag Run
-for each completed interval between 2015-12-01 and 2016-01-02 (but not yet one
for 2016-01-02,
-as that interval hasn't completed) and the scheduler will execute them
sequentially.
+2016-01-02 at 6 AM (or from the command line), with the Airflow 3 default of
+:ref:`CronTriggerTimetable` for ``@daily`` and ``catchup=False``, the scheduler
+does **not** create runs for every midnight since ``start_date``. Instead it
+creates a single Dag run for the most recent applicable tick — midnight on
+**2016-01-02** — with ``data_interval_start`` and ``data_interval_end`` both
+equal to that trigger time. Because that ``run_after`` is already in the past,
+the run can start immediately. The following tick (midnight on 2016-01-03) is
+only created once that schedule time is reached.
+
+If instead the Dag used a data-interval timetable (for example
+:ref:`CronDataIntervalTimetable`, or ``[scheduler]
create_cron_data_intervals=True``),
+the scheduler would immediately create a run for the most recently completed
+interval (2016-01-01 through 2016-01-02), and the next run would cover
+2016-01-02 through 2016-01-03 after that interval ends.
+
+Be aware that using a ``datetime.timedelta`` object as ``schedule`` follows the
+same default-vs-data-interval split via ``[scheduler]
create_delta_data_intervals``.
Review Comment:
Nothing reads `[scheduler] create_delta_data_intervals`. The timedelta
branch of `_create_timetable` gates on `create_cron_data_intervals` too
([dag.py#L149-L152](https://github.com/apache/airflow/blob/20b9a85c557ce0b10e22401a560bf50d856c97a5/task-sdk/src/airflow/sdk/definitions/dag.py#L149-L152)),
so the delta option is inert and this line would send readers to a switch that
does nothing.
Also worth keeping the concrete detail the old paragraph had.
`DeltaMixin._align_to_prev` returns `current` unchanged
([_delta.py#L70-L71](https://github.com/apache/airflow/blob/20b9a85c557ce0b10e22401a560bf50d856c97a5/airflow-core/src/airflow/timetables/_delta.py#L70-L71)),
so with `catchup=False` the first `DeltaTriggerTimetable` run lands at pickup
time (6 AM in this example), not on a midnight tick. That is a bigger
difference from the cron case than "the same split".
##########
airflow-core/docs/core-concepts/dag-run.rst:
##########
@@ -50,24 +50,37 @@ Data Interval
-------------
Each Dag run in Airflow has an assigned "data interval" that represents the
time
-range it operates in. For a Dag scheduled with ``@daily``, for example, each of
-its data interval would start each day at midnight (00:00) and end at midnight
-(24:00).
+range it operates in. How that interval is defined depends on the Dag's
+timetable.
-A Dag run is usually scheduled *after* its associated data interval has ended,
-to ensure the run is able to collect all the data within the time period. In
-other words, a run covering the data period of 2020-01-01 generally does not
-start to run until 2020-01-01 has ended, i.e. after 2020-01-02 00:00:00.
+In Airflow 3, a Dag scheduled with a bare cron string such as ``@daily`` uses
+:ref:`CronTriggerTimetable` by default (``[scheduler]
create_cron_data_intervals``
+is ``False``). For that timetable, ``data_interval_start`` and
+``data_interval_end`` are the same — the trigger time (for ``@daily``, midnight
+each day). The run is created to execute *at* that time.
+
+If you need a contiguous full-day window instead — for example each run
covering
+from midnight to the next midnight — use a data-interval timetable such as
+:ref:`CronDataIntervalTimetable`, or set ``[scheduler]
create_cron_data_intervals``
+to ``True``. With that setup, a Dag run is usually scheduled *after* its
+associated data interval has ended, so a run covering 2020-01-01 generally does
+not start until after 2020-01-02 00:00:00.
+
+See :ref:`Differences between "trigger" and "data interval" timetables` for a
Review Comment:
Nit: a bare `:ref:` pulls in the whole section title, so this renders as
`See Differences between "trigger" and "data interval" timetables for a
side-by-side comparison`, quotes and all. The explicit-title form that the
removed line used reads better:
```
:ref:`timetables comparison <Differences between "trigger" and "data
interval" timetables>`
```
Same for lines 172-173.
##########
airflow-core/docs/core-concepts/dag-run.rst:
##########
@@ -50,24 +50,37 @@ Data Interval
-------------
Each Dag run in Airflow has an assigned "data interval" that represents the
time
-range it operates in. For a Dag scheduled with ``@daily``, for example, each of
-its data interval would start each day at midnight (00:00) and end at midnight
-(24:00).
+range it operates in. How that interval is defined depends on the Dag's
+timetable.
-A Dag run is usually scheduled *after* its associated data interval has ended,
-to ensure the run is able to collect all the data within the time period. In
-other words, a run covering the data period of 2020-01-01 generally does not
-start to run until 2020-01-01 has ended, i.e. after 2020-01-02 00:00:00.
+In Airflow 3, a Dag scheduled with a bare cron string such as ``@daily`` uses
+:ref:`CronTriggerTimetable` by default (``[scheduler]
create_cron_data_intervals``
+is ``False``). For that timetable, ``data_interval_start`` and
+``data_interval_end`` are the same — the trigger time (for ``@daily``, midnight
+each day). The run is created to execute *at* that time.
+
+If you need a contiguous full-day window instead — for example each run
covering
+from midnight to the next midnight — use a data-interval timetable such as
+:ref:`CronDataIntervalTimetable`, or set ``[scheduler]
create_cron_data_intervals``
+to ``True``. With that setup, a Dag run is usually scheduled *after* its
+associated data interval has ended, so a run covering 2020-01-01 generally does
+not start until after 2020-01-02 00:00:00.
+
+See :ref:`Differences between "trigger" and "data interval" timetables` for a
+side-by-side comparison, including how ``logical_date`` and ``run_id`` differ.
All dates in Airflow are tied to the data interval concept in some way. The
"logical date" (also called ``execution_date`` in Airflow versions prior to
2.2)
-of a Dag run, for example, denotes the start of the data interval, not when the
-Dag is actually executed.
+of a Dag run is defined by the timetable: for both timetable kinds it is
+``data_interval_start``. For the default (zero-width) trigger timetable that
+equals the trigger time. With a non-zero ``interval=`` on a trigger timetable
+it is ``trigger_time - interval``. For a data-interval timetable it is the
+start of the contiguous window, not when the Dag is actually executed.
Similarly, since the ``start_date`` argument for the Dag and its tasks points
to
-the same logical date, it marks the start of *the Dag's first data interval*,
not
-when tasks in the Dag will start running. In other words, a Dag run will only
be
-scheduled one interval after ``start_date``.
+the same logical date, it marks the start of scheduling for the Dag (the first
+possible logical date), not when tasks in the Dag will start running. In other
Review Comment:
With a non-zero `interval=` on a trigger timetable, `start_date` is not the
first possible logical date. `restriction.earliest` bounds `run_after`, and the
data interval is `[run_after - interval, run_after]`
([trigger.py#L104-L112](https://github.com/apache/airflow/blob/20b9a85c557ce0b10e22401a560bf50d856c97a5/airflow-core/src/airflow/timetables/trigger.py#L104-L112)),
so `start_date=2015-12-01` with `interval=timedelta(days=1)` gives a first
logical date of 2015-11-30. Since the paragraph above now spells out that case,
"the earliest trigger time" would be more accurate here.
The old "one interval after `start_date`" was also saying something true for
data-interval timetables: the first run does not execute until its window
closes. "Once the timetable reaches `start_date`" loses that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
