Can we do a hybrid of Option 2 and 3?
class DAG:
def __init__(
self,
dag_id: str,
partition: Optional[str] = "auto", # "auto", "strict", "none"
logical_dates: bool = True, # For backward compatibility
):
1. partition_mode="auto": Backward compat, logs warnings when multiple runs
with same logical_date.
2. partition_mode="strict": Can be our traditional method. One run per
logical_date.
3. partition_mode="none": logical_date becomes optional.
For Migrations:
# Legacy DAG - automatically uses strict mode or auto mode with
warnings (*can decide on it later*)
dag = DAG(
'legacy_dag',
schedule_interval='0 0 * * *')
# Modern DAG - explicit non-partitioned approach
dag = DAG(
'modern_dag',
schedule_interval='0 0 * * *',
partition='none',
logical_dates=False)
Thanks,
Avi
On Sat, Dec 21, 2024 at 1:07 AM Mehta, Shubham <shu...@amazon.com.invalid>
wrote:
> Daniel, thank you for bringing this up.
> This is definitely an important discussion, and I encourage others,
> especially Airflow users, to share their thoughts as well. I’ve also shared
> my opinion on the wiki.
>
> Shubham
>
>
