payalbisen-svg opened a new issue, #71187: URL: https://github.com/apache/airflow/issues/71187
### Description [apache/airflow#59115](https://github.com/apache/airflow/pull/59115) (merged in 3.2.0, as part of AIP-76 partition scheduling support) changed how `run_id` is generated for scheduled DAG runs. This is documented plainly in [apache/airflow#65688](https://github.com/apache/airflow/pull/65688): > Airflow 3.2.0 (related PR: #59115) changed how run_id is generated for scheduled DAG runs: > - 3.1.x: `run_id = scheduled__<logical_date>` > - 3.2.0: `run_id = scheduled__<run_after>` This is confirmed intentional, not a bug, see the base `Timetable.generate_run_id()` implementation: https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/timetables/base.py#L460-L474 The issue: `AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS` exists specifically to let cron-scheduled DAGs opt back into Airflow-2-style semantics (logical_date == interval start, run_after offset). It restores `logical_date`/data-interval behavior, but does **not** cover `run_id` generation, `CronDataIntervalTimetable` (the class this setting selects) doesn't override `generate_run_id()`: https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/timetables/interval.py#L133 So users who explicitly opt into `CREATE_CRON_DATA_INTERVALS=True` for full Airflow-2 semantics still end up with `run_id` decoupled from `logical_date`, a partial, easy-to-miss gap in what that setting promises. Related, currently open: [apache/airflow#70167](https://github.com/apache/airflow/pull/70167) is updating `dag-run.rst` to document the new default `run_id`/`logical_date` behavior, confirming this is the accepted, documented state going forward, but doesn't address the `CREATE_CRON_DATA_INTERVALS` coverage gap raised here. What should happen instead: Two possible directions, in rough order of scope: 1. Extend `CREATE_CRON_DATA_INTERVALS` (or a new, separate flag) to also anchor `run_id` to the data interval start when enabled. 2. A more general mechanism e.g. a `dag_run_policy`-style cluster policy hook, that would let users customize `run_id` generation (and potentially other DagRun attributes) without a purpose-built config flag. This was suggested informally by an Airflow core maintainer as possibly the cleaner long-term design, though not something planned near-term. Either would close the gap for deployments that rely on `CREATE_CRON_DATA_INTERVALS` for backward compatibility during 2→3 migrations. Workaround: A custom `Timetable` subclass overriding `generate_run_id()` to key off `data_interval.start` works today and requires no core changes: ```python from airflow.timetables.interval import CronDataIntervalTimetable class LogicalDateRunIdTimetable(CronDataIntervalTimetable): def generate_run_id(self, *, run_type, run_after, data_interval=None, **extra) -> str: if data_interval is not None: return run_type.generate_run_id(suffix=data_interval.start.isoformat()) return super().generate_run_id( run_type=run_type, run_after=run_after, data_interval=data_interval, **extra ) ``` This is DAG-level, not deployment-wide, so it doesn't fully substitute for a config-level fix for users with many DAGs. ### Use case/motivation Airflow deployments migrating from 2 to 3 that rely on `CREATE_CRON_DATA_INTERVALS` for backward compatibility expect that setting to fully restore Airflow-2-style semantics. Currently `run_id` silently falls outside that guarantee, which can break anything downstream that parses `run_id` and assumes it reflects `logical_date`, without any warning that the setting's coverage is partial. ### Related issues _No response_ ### Are you willing to submit a PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
