Pebble32 commented on code in PR #69705:
URL: https://github.com/apache/airflow/pull/69705#discussion_r3657183598
##########
task-sdk/src/airflow/sdk/definitions/timetables/trigger.py:
##########
@@ -86,6 +86,38 @@ class CronTriggerTimetable(CronMixin, BaseTimetable):
run_immediately: bool | datetime.timedelta = attrs.field(kw_only=True,
default=False)
[email protected]
+class JitteredCronTimetable(CronTriggerTimetable):
+ """
+ A :class:`CronTriggerTimetable` that offsets each run by a deterministic,
per-DAG jitter.
+
+ Behaves like ``CronTriggerTimetable`` but shifts every fire time by a
fixed offset derived
+ from ``seed`` and spread across ``[0, max_jitter)``. This avoids the
"thundering herd" where
+ many DAGs sharing a cron expression (e.g. ``@daily`` -> ``0 0 * * *``) all
fire at the same
+ instant and overload the scheduler and workers at that boundary. With the
defaults
+ (``seed=""``, ``max_jitter=0``) the offset is zero, so it behaves exactly
like
+ ``CronTriggerTimetable``.
+
+ The offset is deterministic: the same ``seed`` always maps to the same
offset, so runs stay
+ stable and predictable across scheduler restarts and timetable
serialization. Different seeds
+ land in different slots; collisions are possible but harmless.
``data_interval`` and
+ ``logical_date`` semantics are inherited from ``CronTriggerTimetable`` --
only the wall-clock
+ fire time is shifted.
+
+ :param seed: stable, unique-per-DAG string that determines the offset. The
DAG id is a
+ natural choice.
+ :param max_jitter: upper bound of the jitter window; the offset falls in
``[0, max_jitter)``.
+ Keep it smaller than the gap to the next cron boundary so the shift
cannot push a run's
+ ``logical_date`` across a day or period boundary.
+
+ See :class:`CronTriggerTimetable` for ``cron``, ``timezone``, ``interval``
and
+ ``run_immediately``.
+ """
+
+ seed: str = attrs.field(kw_only=True, default="")
Review Comment:
This is a very good catch didn't think about it thanks! Now raises in both
SDK and core when `max_jitter` > 0 and `seed` is empty, also aligned the
defaults. Added a test for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
